Hot questions for Spring Data for Apache Solr

Top 10 Java Open Source / Spring / Spring Data for Apache Solr

Question:

I am developing a web application using spring MVC and I have hosted this using tomcat. I have a requirement where I want to integrate apache-solr to my search engine.

I tried looking on internet for various tutorials but couldn't find a proper one to explain the steps.

So my requirements are

  1. How to integrate apache solr in spring MVC using maven dependency(I am using tomcat to host the server)
  2. Basic tutorials to add data files with solr
  3. How can I integrate it to pick up the data from my database. I am using hibernate to map my database tables.

After going through some tutorials I think spring-data-solr is suitable for spring MVC.

P.S. I am new to all the above technologies.


Answer:

For Solr in general, you should refer to the Solr reference guide for help. It is meant for the upcoming Solr 5.0 release, but most of the guide is still relevant for 4.0 and the other documentation (the old wiki) is of poor quality.

Solr requires a separate installation; it is not just a maven dependency. To install Solr, follow the install instructions from the guide.

To contact solr in your java layer, you can use solr-j, which you can add as a maven dependency, as noted here, the latest version is 4.10.2:

<dependency> <groupId>org.apache.solr</groupId> <artifactId>solr-solrj</artifactId> <version>4.10.2</version> </dependency>

There's lots of documentation about how to use Solr-J here. spring-data-solr may also work for you.

Hibernate has nothing to do with the integration of solr and spring. Solr can load from your DB directly using the data import handler or you can update documents in solr via its REST API.

Question:

I'm trying to do a query like that retrieves one specific field from the document, i don't a get runtime error when executing the query but i don't get the 3 fields i'm supposed to retrieve from the query, just date and origin, but no the variable, the one that is supposed to return all of them are nulls. How i can select the fields i only want to retrieve in a query?

currently my query looks like this:

  @Query(value = "id:?0", fields = {"?1","date","origin"})
   List<Data> getRecord(String id,String field);

Answer:

Introduction

Reading your comments I see there is a bit of confusion about what's SolrJ and Spring Data for Apache Solr.

SolrJ is the Solr client (personally I would also add the standard and official client).

SolrJ is an API that makes it easy for Java applications to talk to Solr. SolrJ hides a lot of the details of connecting to Solr and allows your application to interact with Solr with simple high-level methods.

Spring Data for Apache Solr is part of the larger framework Spring Data and provides configuration and access to Apache Solr Search Server from Spring applications ( to talk with Solr internally it uses SolrJ).

So far, Solr Spring Data ver. 1.2.0.RELEASE depends on SolrJ 4.7.2 which could be incompatibile with Solr 6.2.0 (for sure if you're using SolrCloud).

Given this appropriate introduction I would suggest to keep Solr instance and SolrJ client on the same version (or at least on the same major version).

So for Solr v.6.2.1 you should use SolrJ 6.2.1, but unfortunately latest Solr Spring Data version is 2.1.3.RELEASE (which internally depends on SolrJ 5.5.0).

<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-solr</artifactId>
        <version>2.1.3.RELEASE</version>
    </dependency>
</dependencies>
Your question

Regarding the fact that you don't receive the list of fields you're looking for, simply Solr Data does not support placeholders for the field list.

Instead of struggling with Solr Spring Data, I would suggest to extend your Repository class creating a new RepositoryImpl where add a custom search using the simple and plain SolrJ client.

@Component
public class ProductsRepositoryImpl implements ProductsRepositoryCustom {

  @Autowired
  private SolrServer   solrServer;

  public ProductsRepositoryImpl() {};

  public ProductsRepositoryImpl(SolrServer solrServer) {
    this.solrServer = solrServer;
  }

  public List<Map<String, Object>> findFields(String id, List<String> fields)
  {
    SolrQuery solrQuery = new SolrQuery("id:" + id);
    solrQuery.setFields(fields.toArray(new String[0]));
    try {
      QueryResponse response = this.solrServer.query(solrQuery);
      return response.getResults()
              .stream()
              .map(d ->
                {
                  return d.entrySet()
                          .stream()
                          .filter(e -> fields.contains(e.getKey()))
                          .collect(Collectors.toMap(p -> p.getKey(), p -> p.getValue()));
                }).collect(Collectors.toList());
    } catch (SolrServerException e) {
      // TODO Auto-generated catch block
    }
    return Collections.emptyList();
  }

}

Question:

I'm writing a Spring MVC based Search API using spring-data-solr 1.0.0.RELEASE with Solr 4.4.0 and Spring 3.2.4.RELEASE.

I'm able to run basic queries but unable to find any good example how can I return score and distance:geodist() in the results.

I know I can get results from Solr using query like

http://localhost:8983/solr/events/select?q=*:*&spatial=true&pt=51.435872%2C-0.427529&sfield=position&d=20&facet=true&facet.mincount=1&facet.limit=-1&facet.field=categoryIds&fl=score,*,distance:geodist()&sort=geodist()+asc

but I don't know how to get score and distance properties returned from Solr using spring-data-solr. I have tried following code

FacetQuery search = new SimpleFacetQuery();

Criteria conditions;// = new Criteria();
if(StringUtils.isNotBlank(searchCriteria.getSearchString()))
        conditions = new Criteria(EventDocument.FIELD_EVENT).contains(searchCriteria.getSearchString());

if(searchCriteria.isLocationKnown()){
        conditions.and(new Criteria(EventDocument.FIELD_POSITION).within(new GeoLocation(searchCriteria.getLatitude(), searchCriteria.getLongitude()), new Distance(searchCriteria.getDistance(), Distance.Unit.MILES)));
        conditions.and(new Criteria(CommonParams.FL).is("score,distance:geodist(),*"));
    } else{
        conditions.and(new Criteria(CommonParams.FL).is("score,*"));
    }

search.addCriteria(conditions);
FacetPage page = solrTemplate.queryForFacetPage(search, EventDocument.class);

but getting following Exception:

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.data.solr.UncategorizedSolrException: undefined field fl; nested exception is org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: undefined field fl
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:948)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

org.springframework.data.solr.UncategorizedSolrException: undefined field fl; nested exception is org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: undefined field fl
org.springframework.data.solr.core.SolrTemplate.execute(SolrTemplate.java:122)
org.springframework.data.solr.core.SolrTemplate.executeSolrQuery(SolrTemplate.java:330)
org.springframework.data.solr.core.SolrTemplate.query(SolrTemplate.java:326)
org.springframework.data.solr.core.SolrTemplate.queryForFacetPage(SolrTemplate.java:285)
com.johnstonpress.api.repository.EventDocumentRepositoryImpl.search(EventDocumentRepositoryImpl.java:72)
com.johnstonpress.api.service.SearchEventServiceImpl.search(SearchEventServiceImpl.java:106)
com.johnstonpress.api.controller.EventsSearchController.searchEvents(EventsSearchController.java:124)
com.johnstonpress.api.controller.EventsSearchController.search(EventsSearchController.java:68)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:601)
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

Thanks for any attention and help!!!


Answer:

To set fl you have to use query.addProjectionOnField(String fieldname).

    SimpleQuery query = new SimpleQuery(conditions);
    query.addProjectionOnField("*");
    query.addProjectionOnField("score");

For mapping score into EventDocument you'll have to add the property as follows.

    @Indexed(value = "score", readonly = true)
    private Float score;

Unfortunately there seems to be an issue with geodist() that might be caused by the way spring data solr creates spatial query. Opended DATASOLR-130 for that.


Distance can be requested using SolrCallback along with SolrTemplate by setting spatial paramters yourself.

    SimpleQuery query = new SimpleQuery(conditions);
    query.addProjectionOnField("*");
    query.addProjectionOnField("distance:geodist()");

    DefaultQueryParser qp = new DefaultQueryParser();
    final SolrQuery solrQuery = qp.constructSolrQuery(query);
    solrQuery.add("sfield", "store");
    solrQuery.add("pt", GeoConverters.GeoLocationToStringConverter.INSTANCE.convert(new GeoLocation(45.15, -93.85)));
    solrQuery.add("d", GeoConverters.DistanceToStringConverter.INSTANCE.convert(new Distance(5)));

    List<EventDocument> result = template.execute(new SolrCallback<List<EventDocument>>() {

      @Override
      public List<EventDocument> doInSolr(SolrServer solrServer) throws SolrServerException, IOException {
        return template.getConverter().read(solrServer.query(solrQuery).getResults(), EventDocument.class);
      }
    });

A litte bit easier would be to provide required parameters for geodist().

    SimpleQuery query = new SimpleQuery(conditions);
    query.addProjectionOnField("*");
    query.addProjectionOnField("distance:geodist(store," + GeoConverters.GeoLocationToStringConverter.INSTANCE.convert(new GeoLocation(45.15, -93.85)) + ")");
    Page<EventDocument> result = template.queryForPage(query, EventDocument.class);

hope that helps!

Question:

I want to write a FacetQuery which may not have any criteria except one filter condition (fq). Following query is an example which I want to build using spring-data-solr API.

http://localhost:8983/solr/jpevents/select?q=*:*&fq=categoryIds:(1101)&facet=true&facet.mincount=1&facet.limit=1&facet.field=primaryCategoryId

How can I set query parameter (q=*:*) in FacetQuery?

Environment: I'm writing a Spring MVC based Search API using spring-data-solr 1.0.0.RELEASE with Solr 4.4.0 and Spring 3.2.4.RELEASE.


Answer:

you can do this combining @Query and @Facet

    @Facet(fields={"primaryCategoryId"}, minCount=1, limit=1)
    @Query(value="*:*", filters="categoryIds:(?0)")
    public FacetPage<JPEvents> XYZ(List<Long> categories, Pageable page);

or execute FacetQuery using SolrTemplate.

   FacetQuery query = new SimpleFacetQuery(new SimpleStringCriteria("*:*"))
     .setFacetOptions(new FacetOptions("primaryCategoryId")
     .setFacetMinCount(1).setFacetLimit(1));
   query.setPageRequest(pageable);
   solrTemplate.queryForFacetPage(query, JPEvents.class);

Question:

I need to implement the following in a Spring Data Solr custom repository:

(X OR Y) AND Z

My current code is as follows:

Criteria criteria = new Criteria("x").is(X_VALUE);
criteria = criteria.or(new Criteria("y").is(Y_VALUE);
criteria = criteria.and(new Criteria("z").is(Z_VALUE);

But running this code I get the following precedence:

X OR (Y AND Z)

Any ideas?


Answer:

The current API does not allow this combination of criteria. There's a patch attached to DATASOLR-105 which could help though it does not solve the problem fully.

Usage of SimpleStringCriteria might help for now. Please be aware of the fact, that values have to be converted to a Solr readable format.

new SimpleQuery(new SimpleStringCriteria("(x or y) and z"));

Update

Version 1.2 will have this issue fixed, so that its possible to create the query as follows.

Criteria orPart = Criteria.where("x").is("foo").or("y").is("bar");
Criteria andPart = Criteria.where("z").is("roo");
Query query = new SimpleQuery(orPart.and(andPart));

Question:

I am trying to implement a Solr Facet search with multi-select on a field. To take this example: http://docs.lucidworks.com/display/solr/Faceting#Faceting-LocalParametersforFaceting, I would like to generate this call to solr:

q=mainquery&fq=status:public&fq={!tag=dt}doctype:pdf&facet=on&facet.field={!ex=dt}doctype

I am not sure how to call this using solrj (java) as I don't want to add a simple filed but I need to include the tagging and excluding (!tag=dt and !ex=dt). Any ideas what the java code should look like?

I am using Spring-Data-Solr which seems to be too basic to make such an advance call. So I think I need to go a level lower and use solrj. Either solution would be great (solrj or spring-data-solr)


Answer:

The following block should create the querystring you want.

SimpleFacetQuery query = new SimpleFacetQuery(new SimpleStringCriteria("mainquery"))
  .addFilterQuery(new SimpleQuery(new Criteria("status").is("public")))
  .addFilterQuery(new SimpleQuery(new Criteria("{!tag=dt}doctype").is("pdf")));
query.setFacetOptions(new FacetOptions("{!ex=dt}doctype"));

solrTemplate.queryForFacetPage(query, YourBean.class);

Question:

I am using Spring Data Solr and I have the following Solr document model class and have a corresponding SolrCrudRepository for this class

@SolrDocument(collection = "oldCollectionName")
public class TestDocument {

            @Id
            @Indexed(name = "id", type = "string")
            private String id;

            @Field(value = "name")
            private String name;

            @Field(value = "externalid")
            private Integer externalId;
}

I am trying to modify the annotation '@SolrDocument(collection = "oldCollectionName")' at runtime.

I have a Service which has the following method to find all documents using the repository and the model class

public List<TestDocument> getDocumentsByName(String name){

        String newSolrDocument = getModifiedSolrCollectionName();
        alterAnnotationValue(TestDocument.class, SolrDocument.class, newSolrDocument);

        SolrDocument solrDocument = TestDocument.class.getAnnotation(SolrDocument.class);

        LOGGER.info("Dynamically set SolrDocument Annotaation: "+solrDocument.collection());

        return testDocumentRepository.findByName(name);
    }

The code to alter annotation looks like this

   public void alterAnnotationValue(Class<?> targetClass, Class<? extends Annotation> targetAnnotation, Annotation targetValue) {
        try {
            Method method = Class.class.getDeclaredMethod(ANNOTATION_METHOD, null);
            method.setAccessible(true);

            Object annotationData = method.invoke(targetClass);

            Field annotations = annotationData.getClass().getDeclaredField(ANNOTATIONS);
            annotations.setAccessible(true);

            Map<Class<? extends Annotation>, Annotation> map = (Map<Class<? extends Annotation>, Annotation>) annotations.get(annotationData);
            map.put(targetAnnotation, targetValue);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

Using this I am correctly getting the newDocumentName set into the annotation map but when calling the testDocumentRepository's find method to find documents. The old collection name is getting picked.

Do I have to do something more for this to work? or I am missing anything?

For reference, I have followed the following tutorial http://www.baeldung.com/java-reflection-change-annotation-params


Answer:

Why don't you just write a custom SolrRepository to solve this? You can inject a SolrTemplate in your custom repository, allowing you to specify a collection for your query like so:

public class TestDocumentRepositoryImpl implements TestDocumentRepository {

    private SolrOperations solrTemplate;
    ...
    public CustomSolrRepositoryImpl(SolrOperations solrTemplate) {
        super();
        this.solrTemplate = solrTemplate;
    }

    @Override
    public TestDocument findOneSpecifyingCollection(String collection, String id) {
        return solrTemplate.getById(collection, id, TestDocument.class);
    }
}

You can do that similarly for repository operation you'd like. People typically need their own implementations if the standard Spring JPA Repositories don't suit their needs. However, you can still mix your own with the standard SolrCrudRepository if that's desirable.

See this for an example from Spring.