How do you debug your Nest queries?

elasticsearch nest get raw query
elasticsearch nest query example
elastic debug query
nest audit trail
nest debug

I'm new to Nest, and I very likely am not creating my query like I think I am. My question is more along the lines of teach a man to fish rather than give me a fish. However, I'll use my current problem as an example.

I have several documents in ElasticSearch of the type Series. I'll stub it out below without the attributes and public modifiers with just the information pertinent to the query:

class Series
{
    string Id {get; set;}
    DateTime StartDate {get; set;}
    DateTime EndDate {get; set;}
    HashSet<Role> ReleasableTo {get; set;}
}

These are all fine and dandy. I can Get() a Series object no problem. The problem I'm running into is trying to figure out how Nest is formatting my query. My immediate goal is to find the most recent Series that is releasable to Role.Visitor. I set up the Nest query like this:

ISearchResponse<Series> response = client
    .Search<Series>(r => 
         r.Filter(f => 
             f.Term<Role>(t=>t.ReleasableTo.First(), Role.Visitor))
    .SortDescending(ser => ser.EndDate).Size(1));

In my mind, this should be producing a query that filters the Series so it only considers the ones that are ReleasableTo my Role.Visitor, reverse sorts by end date, and limits the results to one returned. That would be exactly what I want. In the multiple thousands of records I have for Series, about 90% fit this profile. Unfortunately the query returns 0 results. There is no error, just no results. What I don't know is if I'm using the API incorrectly, if Nest is producing a query structure that doesn't make sense, or I simply don't know ElasticSearch well enough. When I remove the Filter clause I get a result, but I'm not guaranteed everyone is allowed to see it.

How do I view the JSON that Nest produces and sends to ElasticSearch?

You can get the values of search request URL and JSON request body as under:

var requestURL = response.RequestInformation.RequestUrl;
var jsonBody = Encoding.UTF8.GetString(response.RequestInformation.Request);

You can find other useful properties in RequestInformation for debugging.

Debugging, Debuggingedit. When things are going awry, you want to be provided with as much information as possible, to resolve the issue! Elasticsearch.Net and NEST  I rarely nest selectors anymore but when I do, I still usually put the media query directly in the selector. This keeps every selector self-contained and handling its own positioning. Even without nesting selectors, the approach is essentially the same because the media queries go into the selector body.

NEST is Baroque of .NET APIs. For 2.1+ on call level:

IElasticClient client = new ElasticClient();
var searchDescriptor = new SearchDescriptor<Series>();
var query = Query<Series>.Term(...);
var pretty = query.ToPrettyString(query);
var json = client.ToRawRequest(searchDescriptor.Query(descriptor => query));

On configuration level:

    var settings = new ConnectionSettings()
                     .PrettyJson().DisableDirectStreaming()
                     .OnRequestCompleted(details=> Debug.WriteLine(Encoding.UTF8.GetString(details.RequestBodyInBytes)));

On response level look into CallDetails.RequestBodyInBytes.

Used extensions:

    /// <summary>
    /// Converts search to raw JSON request for debugging.
    /// </summary>
    /// <typeparam name="T">The type.</typeparam>
    /// <param name="self">The self.</param>
    /// <param name="searchDescriptor">The search descriptor.</param>
    /// <returns>The string.</returns>
    public static string ToRawRequest<T>(this IElasticClient self, SearchDescriptor<T> searchDescriptor) where T : class
    {
        using (var output = new MemoryStream())
        {
            self.Serializer.Serialize(searchDescriptor, output);
            output.Position = 0;
            var rawQuery = new StreamReader(output).ReadToEnd();
            return rawQuery;
        }
    }

    /// <summary>
    /// Prints query into string.
    /// </summary>
    /// <param name="self">The self.</param>
    /// <returns>The value.</returns>
    public static string ToPrettyString(this QueryContainer self)
    {
        using (var settings = new ConnectionSettings())
        {
            var visitor = new DslPrettyPrintVisitor(settings);
            self.Accept(visitor);
            return visitor.PrettyPrint.Replace(Environment.NewLine, string.Empty);
        }                                                                         
    }

Debug information, Select queries (as well as insert, delete, and update statements) can be nested. We separate the levels of nesting, testing logic from the inside out  When things are going awry, you want to be provided with as much information as possible, to resolve the issue! Elasticsearch.Net and NEST provide an audit trail and debug information to help get you back on the happy path.

I like to take it a step further than bsarkar suggests and eliminate the need for a roundtrip altogether:

var client = new ElasticClient();

var seriesSearch = new SearchDescriptor<Series>();
seriesSearch.Filter(f => f
    .Term<Role>(t => t.ReleasableTo.First(), Role.Visitor))
    .SortDescending(ser => ser.EndDate)
    .Size(1));

string searchJson = Encoding.UTF8.GetString(client.Serializer.Serialize(seriesSearch));

Note that your ElasticClient doesn't need any connection properties, so you have no dependency on an ES node.

Debugging Complex SQL Queries, I believe something like this exists in debug, but not for use under normal Hey @rianjs, you can actually serialize your NEST query object to  How do you debug your Nest queries? Filter items which array contains any of given values ; How to disable camel casing Elasticsearch field names in NEST? How do I update an existing document inside ElasticSearch index using NEST? is there a way to deserialize Elasticsearch Nest search query?

Really easily. If this is my code that searches:

var results = client.Search<SearchItem>(s => s.AllIndices()
    .Query(q =>
            q.Term(p => p.LastName, searchItem.LastName)
            && q.Term(p => p.FirstName, searchItem.FirstName)
            && q.Term(p => p.ApplicationCode, searchItem.ApplicationCode)
            )
    .Size(1000)
    );
var list = results.Documents.ToList();

I then set a breakpoint on the line above. Then, in Visual Studio Immediate Window, I enter this:

?results.ConnectionStatus

and it gives me this:

{StatusCode: 200, 
    Method: POST, 
    Url: http://localhost:9200/_all/searchitem/_search, 
    Request: {
  "size": 1000,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "lastName": {
              "value": "carr"
            }
          }
        },
        {
          "term": {
            "firstName": {
              "value": "adrian"
            }
          }
        }
      ]
    }
  }
}

Hope this helps.

Support for getting the raw query JSON without sending it to ES , Map example where “map” is a nested HashMap object. SELECT To set debugging on the query level, add the <trace> keyword before the query. (If you are  This degrades rather rapidly and makes inspecting generated queries quite a chore. NEST does its best to detect these cases and will always write them in the first, cleaner form. Conditionless Queriesedit. Writing complex boolean queries is one thing, but more often then not you’ll want to make decisions on how to query based on user input.

Using the latest elastic search 5+, I was able to get mine (thanks to Adrian Carr's method) with the following:

var jsonOutput = System.Text.Encoding.UTF8.GetString(
    response.ApiCall.RequestBodyInBytes
)

Which gave me the following output:

{
   "from":0,
   "size":0,
   "query":{
      "bool":{
      ...
      }
   }
}

GemFire Querying FAQ and Examples, Debugging a query plan almost always follows the same pattern: take time to The strategy Postgres has chosen for our query is nested loop, the most naive of​  c# - How do you debug your Nest queries? I'm new to Nest, and I very likely am not creating my query like I think I am. My question is more along the lines of teach a man to fish rather than give me a fish. However, I'll use my current pro…

Debugging the Postgres query planner, The FROM term is a keyspace reference or expression which defines the input object(s) for the query, representing the left-hand side of the NEST clause. If you answer one of these questions with No, you are a really good query builder or you never build queries. If you answer one of these questions with Yes, this blog post can be something for you. Within SQL 2008 a debug option is introduced in the SQL Server Management Studio (SSMS).

NEST Clause, To disable logging, set the logger property to false in the (optional) Nest Values in the array can be any combination of 'log' , 'error' , 'warn' , 'debug' , and  Re: Logging of queries for debugging May be changing log level in logging.yml will show you some elements. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 3 août 2013 à 01:07, "Oliver B. Fischer" < [hidden email] > a écrit : Hi David, I set all settings to 1ms but ES logs only that a slow search happened but not the query itself.

Logger, For example, it might be decided to use a sort-merge join or a nested-loop join for a particular join operator. These two parts of query optimization are often. FilterContainer no longer exists in NEST 2.0 onwards, because filters and queries merged in Elasticsearch into queries, with a query or filter context. Where you would have used FilterContainer, use QueryContainer and then when adding to the search, pass the collection of FilterContainer into a bool query filter clause.

Comments
  • For future readers, take a look at the documentation on logging and debugging: elastic.co/guide/en/elasticsearch/client/net-api/current/…
  • That's very helpful, for what it's worth, if I do a "query": {"match":{"releasableTo": "Visitor")) I get what I want. I think my index needs some fine tuning.
  • PrettyJson, DisableDirectStreaming, and OnRequestCompleted can all be done in one call with EnableDebugMode in newer versions of nest. See my answer below for example.
  • Note, in NEST 2.4.5 the Serialize function no longer returns a string and must take a target Stream as a parameter. e.g. using (var searchRequestStream = new MemoryStream()) { client.Serializer.Serialize(seriesSearch, searchRequestStream); var searchRequestString = Encoding.UTF8.GetString(searchRequestStream.GetBuffer()); }
  • This is a really useful technique.
  • Not sure what I am missing, but I see no ConnectionStatus property on my results object