Hot questions for Spring Batch Admin

Hot questions for Spring Batch Admin

Top 10 Java Open Source / Spring / Spring Batch Admin

Question:

I have an application which uses Spring Batch and Spring MVC. I am able to deploy Spring Batch Admin as a separate war and use it against the same DB my application uses, though I would like to integrate it into my own application, possibly modify some of the views as well.

Is there an easy way to do this or do I have to fork it and go from there?


Answer:

There is an easy way apparently according to this thread;

  • Define a DispatcherServlet for Batch Admin in web.xml:

    <servlet>
        <servlet-name>Batch Servlet</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath*:/org/springframework/batch/admin/web/resources/servlet-config.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
    
    <servlet-mapping>
        <servlet-name>Batch Servlet</servlet-name>
        <url-pattern>/batch/*</url-pattern>
    </servlet-mapping>
    
  • Add an override for resourceService in the root appContext:

    <bean id="resourceService"
    class="org.springframework.batch.admin.web.resources.DefaultResourceService">
        <property name="servletPath" value="/batch" />
    </bean> 
    
  • Modify standard.ftl in spring-batch-admin-resources-1.2.0-RELEASE.jar to reflect the URL:

    <#assign url><@spring.url relativeUrl="${servletPath}/resources/styles/main.css"/></#assign>

Question:

I want to create a flat file which has the below format:

Col1Name;Col2Name;Col3Name
one;23;20120912
two;28;20120712

As seen, the first line in the flat file are the column names.

How to achieve this through header callback ?

I see that if the input file is of above format, there is an option as below to ignore first line:

<property name="firstLineIsHeader" value="true"/>

Also, this Jira Issue indicates that what I want is implemeted and closed. However, I am unable to find any example for writing first line as column names.

<beans:bean id="MyFileItemWriter" class="com.nik.MyFileItemWriter" scope="step">
    <beans:property name="delegate">
        <beans:bean class="org.springframework.batch.item.file.FlatFileItemWriter">
            <beans:property name="resource" value="file:MYFILE.dat" /> 

            <beans:property name="lineAggregator">
                <beans:bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
                    <beans:property name="delimiter" value=";" />
                    <beans:property name="fieldExtractor">
                        <beans:bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
                            <beans:property name="names" value="Col1Name, Col2Name, Col3Name" />
                        </beans:bean>
                    </beans:property>
                </beans:bean>
            </beans:property>
            <beans:property name="headerCallback" ref="MyFileItemWriter" />
        </beans:bean>
    </beans:property>
</beans:bean>

My Item Writer looks as below:

public class MyFileItemWriter implements ItemWriter<MyBean>, FlatFileHeaderCallback, ItemStream{

private FlatFileItemWriter<MyBean> delegate;    

 public void setDelegate(final FlatFileItemWriter<MyBean> delegate) {
        this.delegate = delegate;
    }

public void writeHeader(Writer writer) throws IOException {


}

public void write(List<? extends MyBean> items) throws Exception {
    this.delegate.write(items);

}

public void close() throws ItemStreamException {
     this.delegate.close();

}

public void open(ExecutionContext arg0) throws ItemStreamException {
     this.delegate.open(arg0);

}

public void update(ExecutionContext arg0) throws ItemStreamException {
     this.delegate.update(arg0);        
}

}

Thanks for reading!


Answer:

create a custom class which extends the FlatFileItemWriter and implements just the constructor:

public class MyFlatFileWriter extends FlatFileItemWriter {

    public MyFlatFileWriter (){
        super.setHeaderCallback(new FlatFileHeaderCallback() {

            public void writeHeader(Writer writer) throws IOException {
                writer.write("Col1Name,Col2Name,Col3Name");

            }
        });
    }

and then use this class in the bean configuration class attribute

Question:

I am processing CSV files using FlatFileItemReader.

Sometimes I am getting blank lines within the input file.

When that happened the whole step stops. I want to skipped those lines and proceed normal.

I tried to add exception handler to the step in order to catch the execption instead of having the whole step stooped:

@Bean
    public Step processSnidUploadedFileStep() {
        return stepBuilderFactory.get("processSnidFileStep")
                .<MyDTO, MyDTO>chunk(numOfProcessingChunksPerFile) 
                .reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
                .processor(manualUploadAsyncItemProcessor())
                .writer(manualUploadAsyncItemWriter())
                .listener(logProcessListener)
                .throttleLimit(20)
                .taskExecutor(infrastructureConfigurationConfig.taskJobExecutor())
                .exceptionHandler((context, throwable) -> logger.error("Skipping record on file. cause="+ ((FlatFileParseException)throwable).getCause()))
                .build();
    }

Since I am processing with chunks when blank line arrives and exception is catched what's happens is that the whole chunk is skipped(the chunk might contains valid lines on CSV file and they are skipped aswell)

Any idea how to do this right when processing file in chunks?

Thanks, ray.

After editing my code. still not skipping:

public Step processSnidUploadedFileStep() {
        SimpleStepBuilder<MyDTO, MyDTO> builder = new SimpleStepBuilder<MyDTO, MyDTO>(stepBuilderFactory.get("processSnidFileStep"));
       return builder
                .<PushItemDTO, PushItemDTO>chunk(numOfProcessingChunksPerFile)
                .faultTolerant().skip(FlatFileParseException.class)
                .reader(snidFileReader(OVERRIDDEN_BY_EXPRESSION))
                .processor(manualUploadAsyncItemProcessor())
                .writer(manualUploadAsyncItemWriter())
                .listener(logProcessListener)
                .throttleLimit(20)
                .taskExecutor(infrastructureConfigurationConfig.taskJobExecutor())
                .build();
    }

Answer:

We created custom SimpleRecordSeparatorPolicy which is telling reader to skip blank lines. That way we read 100 records, i.e. 3 are blank lines and those are ignored without exception and it writes 97 records.

Here is code:

package com.my.package;

import org.springframework.batch.item.file.separator.SimpleRecordSeparatorPolicy;

public class BlankLineRecordSeparatorPolicy extends SimpleRecordSeparatorPolicy {

    @Override
    public boolean isEndOfRecord(final String line) {
        return line.trim().length() != 0 && super.isEndOfRecord(line);
    }

    @Override
    public String postProcess(final String record) {
        if (record == null || record.trim().length() == 0) {
            return null;
        }
        return super.postProcess(record);
    }

}

And here is reader:

package com.my.package;

import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.stereotype.Component;

@Component
@StepScope
public class CustomReader extends FlatFileItemReader<CustomClass> {

    @Override
    public void afterPropertiesSet() throws Exception {
        setLineMapper(new DefaultLineMapper<CustomClass>() {
            {
                /// configuration of line mapper
            }
        });
        setRecordSeparatorPolicy(new BlankLineRecordSeparatorPolicy());
        super.afterPropertiesSet();
    }
}

Question:

When using Spring Batch Admin, it tries to provide some defaults for dataSource, transactionManager etc.

If you want to override these defaults, you create your own xml bean definitions under META-INF/spring/batch/servlet/override/ folder and during the bootstrap it guarantees that the default properties will be overridden.

In spring-batch-admin, a dataSource default is defined in data-source-context.xml with this definition

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" value="${batch.jdbc.driver}" />
    <property name="url" value="${batch.jdbc.url}" />
    <property name="username" value="${batch.jdbc.user}" />
    <property name="password" value="${batch.jdbc.password}" />
    <property name="testWhileIdle" value="${batch.jdbc.testWhileIdle}"/>
    <property name="validationQuery" value="${batch.jdbc.validationQuery}"/>
</bean>

Now, I want to override this dataSource with a JNDI datasource so I removed the property lines like batch.jdbc.driver, batch.jdbc.url and have the following jndi definition

<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
     <property name="jndiName" value="java:comp/env/jdbc/dbconn" />
</bean> 

As you may easily guess the system first tries to initialize the dataSource bean defined in data-source-context.xml and since it cannot find any values for property values batch.jdbc.* it fails with an exception.

Could not resolve placeholder 'batch.jdbc.driver' in string value [${batch.jdbc.driver}]

Since I will be using JNDI and do not want to deal with these property values, I cannot proceed.

Any idea on how to override dataSource in this situation?


Answer:

Within Spring Batch Admin, there are 2 Spring ApplicationContexts that are being loaded:

  • servlet-config.xml
  • webapp-config.xml

servlet-config.xml has these imports:

<import resource="classpath*:/META-INF/spring/batch/servlet/resources/*.xml" />
<import resource="classpath*:/META-INF/spring/batch/servlet/manager/*.xml" />
<import resource="classpath*:/META-INF/spring/batch/servlet/override/*.xml" />

webapp-config.xml has these imports:

<import resource="classpath*:/META-INF/spring/batch/bootstrap/**/*.xml" />
<import resource="classpath*:/META-INF/spring/batch/override/**/*.xml" />

servlet-config.xml configurers the servlet, webapp-config.xml configures (the backend part of the) the application. The problem is that the dataSource bean is part of/defined in the second config, not the first. Hence, when you add the dataSource bean to an override for the servlet config(/META-INF/spring/batch/servlet/override/*.xml), as you are doing, you add a new bean to the first context, instead of overwriting the dataSource bean of the second context.

So, you need to put your custom data-source-context.xml under META-INF/spring/batch/override/ instead of META-INF/spring/batch/servlet/override/

Then it works and you won't even get the Could not resolve placeholder 'batch.jdbc.driver' in string value [${batch.jdbc.driver}] error.

Question:

I have 2 different jobs (actually more but for simplicity assume 2). Each job can run in parallel with the other job, but each instance of the same job should be run sequentially (otherwise the instances will cannibalize eachother's resources).

Basically I want each of these jobs to have it's own queue of job instances. I figured I could do this using two different thread pooled job launchers (each with 1 thread) and associating a job launcher with each job.

Is there a way to do this that will be respected when launching jobs from the Spring Batch Admin web UI?


Answer:

There is a way to specify a specific job launcher for a specific job, but the only way I have found to do it is through use of a JobStep.

If you have a job called "specificJob" this will create another job "queueSpecificJob" so when you launch it, either through Quartz or Spring Batch web admin, it will queue up a "specificJob" execution.

<bean id="specificJobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository"/>
    <property name="taskExecutor">
        <task:executor id="singleThreadPoolExecutor" pool-size="1"/>
    </property>
</bean>

<job id="queueSpecificJob">
    <step id="specificJobStep">
        <job ref="specificJob" job-launcher="specificJobLauncher" job-parameters-extractor="parametersExtractor" />
    </step>
</job>

Question:

I am using spring MVC. From my controller, I am calling jobLauncher and in jobLauncher I am passing job parameters like below and I'm using annotations to enable configuration as below:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
        // read, write ,process and invoke job
} 

JobParameters jobParameters = new JobParametersBuilder().addString("fileName", "xxxx.txt").toJobParameters();
stasrtjob = jobLauncher.run(job, jobParameters);                              

and here is my itemprocessor                                                         
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {

  public OutPutData process(final InputData inputData) throws Exception {

        // i want to get job Parameters here ????

  }

}

Answer:

1) Put a scope annotation on your data processor i.e.

@Scope(value = "step") 

2) Make a class instance in your data processor and inject the job parameter value by using value annotation :

@Value("#{jobParameters['fileName']}")
private String fileName;

Your final Data processor class will look like:

@Scope(value = "step")
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {

@Value("#{jobParameters['fileName']}")
private String fileName;

  public OutPutData process(final InputData inputData) throws Exception {

        // i want to get job Parameters here ????
      System.out.println("Job parameter:"+fileName);

  }

  public void setFileName(String fileName) {
        this.fileName = fileName;
    }


}

In case your data processor is not initialized as a bean, put a @Component annotation on it:

@Component("dataItemProcessor")
@Scope(value = "step")
public class DataItemProcessor implements ItemProcessor<InputData, OutPutData> {

Question:

I have a spring batch application which has the property file batch-default.properties set up as

batch.job.configuration.file.dir=target/config

Now this application works well on my local machine even though i do not have any such directory but when i try to deploy the same on my integration server i am getting the error:

Cannot resolve reference to bean 'org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0.source' while setting bean property 'source'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0.source': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: Source directory [target/config] does not exist.
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:334)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1417)

Has anyone faced any similar problem?

Any help appriciated here.

-Vaibhav


Answer:

We had a similar issue on our test server. Looks like a problem with file permissions, i.e. Spring Batch Admin tries to create a directory structure for "target/config", but the user is not allowed to create the directories.

This is the chain causing the problem:

META-INF/spring/batch/bootstrap/integration/configuration-context.xml contains the definition of a file poller refering to the property:

<file:inbound-channel-adapter directory="${batch.job.configuration.file.dir}" channel="job-configuration-files"
        filename-pattern=".*\.xml">
        <poller max-messages-per-poll="1" cron="5/1 * * * * *" />
    </file:inbound-channel-adapter>

Checking the documentation of the inbound-channel-adapter reveals the following (spring-integration-file-2.1.xsd):

 <xsd:attribute name="directory" type="xsd:string" use="required">
                <xsd:annotation>
                    <xsd:documentation><![CDATA[Specifies the input directory (The directory to poll from) e.g.:
                    directory="file:/absolute/input" or directory="file:relative/input"]]></xsd:documentation>
                </xsd:annotation>            
            </xsd:attribute>

and (!)

 <xsd:attribute name="auto-create-directory" type="xsd:string" default="true">
                <xsd:annotation>
                    <xsd:documentation>
                        Specify whether to automatically create the source directory if it does not yet exist when this
                        adapter is being initialized. The default value is 'true'. If set to 'false' and the directory
                        does not exist upon initialization, an Exception will be thrown.
                    </xsd:documentation>
                </xsd:annotation>
            </xsd:attribute>

So, as an result, auto-create-directory is true and Spring is trying to create the (relative) directory structure somewhere on the shell path of your server.

For the error message, checking the java class org.springframework.integration.file.FileReadingMessageSource gives the explanation:

if (!this.directory.exists() && this.autoCreateDirectory) {
            this.directory.mkdirs();
        }
        Assert.isTrue(this.directory.exists(),
                "Source directory [" + directory + "] does not exist.");

The javadoc of java.io.File.mkdirs() says:

public boolean mkdirs()

Creates the directory named by this abstract pathname, including any necessary but nonexistent parent directories. Note that if this operation fails it may have succeeded in creating some of the necessary parent directories.

Returns:
    true if and only if the directory was created, along with all necessary parent directories; false otherwise

So what happens is, the mkdirs() returns "false" because he couldn't create the directory. The following exists() will also return "false" returning the error messsage as stated in the original post.

You can workaround be setting the parameter to an existing and writeable directory like "/tmp" using an absolute path. Unfortunately I have no idea how this spring batch feature should work, if you store your job definitions on a classpath; it would make more sense not to use a file poller but to use a "classpath-aware" file poller...