Hot questions for Using Vapor in future

Question:

I'm struggling with understanding how to perform a batch save of fetched objects and store them to the database. After I have stored the objects to the database I want to return the result of a query. I can't understand how to do this with EventLoopFuture since when I call .wait() I get the error message:

Precondition failed: BUG DETECTED: wait() must not be called when on an EventLoop.

As an example of my problem:

  • I need to fetch an entity from an external endpoint (let's say flights for an airport)
  • The result of that call needs to be saved to a database. If the flight exists in the database, it needs to be updated otherwise created.
  • When completed, a list of all the flights in from the database needs to be returned.

This is what I got so far, but which gives me the bug:

func flights(on conn: DatabaseConnectable, customerName: String, flightType: FlightType) throws -> Future<[Flight]> {

    return Airport.query(on: conn).filter(\.customerName == customerName).first().flatMap(to: [Flight].self) { airport in
      guard let airport = airport else {
        throw Abort(.notFound)
      }

      guard let airportId = airport.id else {
        throw Abort(.internalServerError)
      }

      // Update items for customer
      let fetcher: AirportManaging?

      switch customerName.lowercased() {
      case "coolCustomer":
        fetcher = StoreOneFetcher()
      default:
        fetcher = nil
        debugPrint("Unhandled customer to fetch from!")
        // Do nothing
      }

      let completion = Flight.query(on: conn).filter(\.airportId == airportId).filter(\.flightType == flightType).all

      guard let flightFetcher = fetcher else { // No customer fetcher to get from, but still return whats in the DB
        return completion()
      }

      return try flightFetcher.fetchDataForAirport(customerName, on: conn).then({ (flights) -> EventLoopFuture<[Flight]> in
        flights.forEach { flight in
          _ = try? self.storeOrUpdateFlightRecord(flight, airport: airport, on: conn).wait()
        }
        return completion()
      })
    }
  }

  func storeOrUpdateFlightRecord(_ flight: FetcherFlight, airport: Airport, on conn: DatabaseConnectable) throws -> EventLoopFuture<Flight> {
    guard let airportId = airport.id else {
      throw Abort(.internalServerError)
    }

    return Flight.query(on: conn).filter(\.itemName == flight.itemName).filter(\.airportId == airportId).filter(\.flightType == flight.type).all().flatMap(to: Flight.self) { flights in
      if let firstFlight = flights.first {
        debugPrint("Found flight in database, updating...")
        return flight.toFlight(forAirport: airport).save(on: conn)
      }

      debugPrint("Did not find flight, saving new...")
      return flight.toFlight(forAirport: airport).save(on: conn)
    }
  }

So the problem in on line _ = try? self.storeOrUpdateFlightRecord(flight, airport: airport, on: conn).wait(). I can't call wait() since it'll block the eventLoop but if I call map or flatMap I need in turn return an EventLoopFuture<U> (U being Flight), which I am totally uninterested in.

I want self.storeOrUpdateFlightRecord to be called and the result ignored. How can I do that?


Answer:

Yeah, you can't use .wait() on eventLoop.

In your case you could use flatten for batch operations

/// Flatten works on array of Future<Void>
return flights.map {
    try self.storeOrUpdateFlightRecord($0, airport: airport, on: conn)
        /// so transform a result of a future to Void
        .transform(to: ())
}
/// then run flatten, it will return Future<Void> as well
.flatten(on: conn).flatMap {
    /// then do what you want :)
    return completion()
}

Question:

I'm using Vapor 3 and linking to a FoundationDB database, so Im not using Fluent. I have a method that searches for a record, but it will obviously crash if it does not return a record (because I force unwrap the value).

I want to guard the reading from the database and return a response if no record is found. This will however not be the record as was expected in the Future. I was thinking that I should return a different response, but am unsure how to change the result that is expected.

//creates a specific country
func getCountry( req: Request) throws -> Future<Country> {
    // get Country name from get parameter string
    let countryString = try req.parameters.next(String.self)


    // get record from Database. This could fail and so needs to be guarded. What response should be returned as the Future requires a Country datatype?

       let record =  FDBConnector().getRecord(path: Subspace("iVendor").subspace(["Countries", countryString]))



    let newCountry = try JSONDecoder().decode(Country.self, from: record!)
    // return Country Struct
    return Future.map(on: req) {return newCountry }

}

Answer:

There are a couple of options here.

First, if you throw an error from the method:

guard let record =  FDBConnector().getRecord(path: Subspace("iVendor").subspace(["Countries", countryString])) else {
    throw Abort(.notFound, reason: "No country found with name \(countryString)")
}

The error will get converted to a 404 (Not Found) response with "No country found with name \(countryString)" as the error message.

If you want more control over the resulting response, you can change the route hander's return type to Future<Response>. You can then encode the Country object to the response or create a custom error response. This method does take some extra work though.

let response = Response(using: req)
guard let record =  FDBConnector().getRecord(path: Subspace("iVendor").subspace(["Countries", countryString])) else {
    try response.content.encode(["message": "Country not found"])
    response.http.status = .notFound
    return response
}

try response.content.encode(record)
return response

Note that you will have to conform Country to Content if you want that snippet to work.

Question:

I tried to make the most basic example that I could think of for my problem. I have a Course model and a many-to-many table to User that also stores some extra properties (the progress in the example below).

import FluentPostgreSQL
import Vapor

final class Course: Codable, PostgreSQLModel {
  var id: Int?
  var name: String
  var teacherId: User.ID

  var teacher: Parent<Course, User> {
    return parent(\.teacherId)
  }

  init(name: String, teacherId: User.ID) {
    self.name = name
    self.teacherId = teacherId
  }
}

struct CourseUser: Pivot, PostgreSQLModel {
  typealias Left = Course
  typealias Right = User

  static var leftIDKey: LeftIDKey = \.courseID
  static var rightIDKey: RightIDKey = \.userID

  var id: Int?
  var courseID: Int
  var userID: UUID
  var progress: Int

  var user: Parent<CourseUser, User> {
    return parent(\.userID)
  }
}

Now, when I return a Course object, I want the JSON output to be something like this:

{
  "id": 1,
  "name": "Course 1",
  "teacher": {"name": "Mr. Teacher"},
  "students": [
    {"user": {"name": "Student 1"}, progress: 10},
    {"user": {"name": "Student 2"}, progress: 60},
  ]
}

Instead of what I would normally get, which is this:

{
  "id": 1,
  "name": "Course 1",
  "teacherID": 1,
}

So I created some extra models and a function to translate between them:

struct PublicCourseData: Content {
  var id: Int?
  let name: String
  let teacher: User
  let students: [Student]?
}

struct Student: Content {
  let user: User
  let progress: Int
}

extension Course {
  func convertToPublicCourseData(req: Request) throws -> Future<PublicCourseData> {
    let teacherQuery = self.teacher.get(on: req)
    let studentsQuery = try CourseUser.query(on: req).filter(\.courseID == self.requireID()).all()

    return map(to: PublicCourseData.self, teacherQuery, studentsQuery) { (teacher, students) in
      return try PublicCourseData(id: self.requireID(),
                                  name: self.name,
                                  teacher: teacher,
                                  students: nil) // <- students is the wrong type!
    }
  }
}

Now, I am almost there, but I am not able to convert studentsQuery from EventLoopFuture<[CourseUser]> to EventLoopFuture<[Student]>. I tried multiple combinations of map and flatMap, but I can't figure out how to translate an array of Futures to an array of different Futures.


Answer:

The logic you're looking for will look like this

extension Course {
    func convertToPublicCourseData(req: Request) throws -> Future<PublicCourseData> {
        return teacher.get(on: req).flatMap { teacher in
            return try CourseUser.query(on: req)
                                 .filter(\.courseID == self.requireID())
                                 .all().flatMap { courseUsers in
                // here we should query a user for each courseUser
                // and only then convert all of them into PublicCourseData
                // but it will execute a lot of queries and it's not a good idea
            }
        }
    }
}

I suggest you to use the SwifQL lib instead to build a custom query to get needed data in one request 🙂

You could mix Fluent's queries with SwifQL's in case if you want to get only one course, so you'll get it in 2 requests:

struct Student: Content {
    let name: String
    let progress: Int
}

extension Course {
    func convertToPublicCourseData(req: Request) throws -> Future<PublicCourseData> {
        return teacher.get(on: req).flatMap { teacher in
            // we could use SwifQL here to query students in one request
            return SwifQL.select(\CourseUser.progress, \User.name)
                        .from(CourseUser.table)
                        .join(.inner, User.table, on: \CourseUser.userID == \User.id)
                        .execute(on: req, as: .psql)
                        .all(decoding: Student.self).map { students in
                return try PublicCourseData(id: self.requireID(),
                                          name: self.name,
                                          teacher: teacher,
                                          students: students)
            }
        }
    }
}

If you want to get a list of courses in one request you could use pure SwifQL query.

I simplified desired JSON a little bit

{
  "id": 1,
  "name": "Course 1",
  "teacher": {"name": "Mr. Teacher"},
  "students": [
    {"name": "Student 1", progress: 10},
    {"name": "Student 2", progress: 60},
  ]
}

first of all let's create a model to be able to decode query result into it

struct CoursePublic: Content {
    let id: Int
    let name: String
    struct Teacher:: Codable {
        let name: String
    }
    let teacher: Teacher
    struct Student:: Codable {
        let name: String
        let progress: Int
    }
    let students: [Student]
}

Ok now we are ready to build a custom query. Let's build it in some request handler function

func getCourses(_ req: Request) throws -> Future<[CoursePublic]> {
    /// create an alias for student
    let s = User.as("student")

    /// build a PostgreSQL's json object for student
    let studentObject = PgJsonObject()
        .field(key: "name", value: s~\.name)
        .field(key: "progress", value: \CourseUser.progress)

    /// Build students subquery
    let studentsSubQuery = SwifQL
        .select(Fn.coalesce(Fn.jsonb_agg(studentObject),
                            PgArray(emptyMode: .dollar) => .jsonb))
        .from(s.table)
        .where(s~\.id == \CourseUser.userID)


    /// Finally build the whole query
    let query = SwifQLSelectBuilder()
        .select(\Course.id, \Course.name)
        .select(Fn.to_jsonb(User.table) => "teacher")
        .select(|studentsSubQuery| => "students")
        .from(User.table)
        .join(.inner, User.table, on: \Course.teacherId == \User.id)
        .join(.leftOuter, CourseUser.table, on: \CourseUser.teacherId == \User.id)
        .build()
    /// this way you could print raw query
    /// to execute it in postgres manually
    /// for debugging purposes (e.g. in Postico app)
    print("raw query: " + query.prepare(.psql).plain)
    /// executes query with postgres dialect
    return query.execute(on: req, as: .psql)
        /// requests an array of results (or use .first if you need only one first row)
        /// You also could decode query results into the custom struct
        .all(decoding: CoursePublic.self)
}

Hope it will help you. There may be some mistakes in the query cause I wrote it without checking 🙂 You can try to print a raw query to copy it and execute in e.g. Postico app in postgres directly to understand what's wrong.

Question:

How to return user if based on facebook user ID it already exist, and create a new user if not exist in Vapor? You can see how I tried fetch data, but get error.

final class User: Content {
    var id: Int?
    var fbId: String

    init(id: Int? = nil, fbId: String) {
        self.id = id
        self.fbId = fbId
    }
}

router.get("user") { (request) -> Future<User> in
    return Future.map(on: request) { () -> User in
        let fbId = try request.query.get(String.self, at: "fbId")
        return User.query(on: request).filter(\.fbId == fbId).first().map { (user) -> (U) in
            if user == nil {
                user = User(fbId: fbId)
            }
            return user
        }
    }
}


Answer:

You have a few things going on here. To start with you don't need the first Future.map - not sure what that's doing.

Then you have the issue of the compiler - you have to return the same type in each closure and the function, which is awkward because if you already have a user you can return that, if you don't you need to create and save one, which returns Future<User>, which is not the same to User.

So to answer your question, U there should be User, but really you want to change first().map to first().flatMap in which case U becomes Future<User>. Then you can do something like:

router.get("user") { req -> Future<User> in
    let fbID = try req.query.get(String.self, at: "fbId")
    return User.query(on: req).filter(\.fbId == fbID).first().flatMap { user in
        let returnedUser: Future<User>
        if let foundUser = user {
            returnedUser = req.future(foundUser)
        } else {
            let newUser = User(fbId: fbID)
            returnedUser = newUser.save(on: req)
        }
        return returnedUser
    }
}

To solve your problems. Hope that helps!

Question:

Have following code:

router.get("/fetchOngoingReleases", String.parameter) { (request) -> Future<[ReleaseWithUser]> in
    return Release.query(on: request).filter(\.inProgress == true).all().flatMap { (ra) -> EventLoopFuture<[ReleaseWithUser]> in
        let userId = try request.parameters.next(String.self)
        return User.query(on: request).filter(\.fbId == userId).first().flatMap { (user) -> EventLoopFuture<[ReleaseWithUser]> in
            let a = ra.map { (r) -> ReleaseWithUser in
                // some condition and logic here
            }

            return a. //need this to make future
        }
    }
}

I checked all the possible method calls but none of converts [ReleaseWithUser] to Future<[ReleaseWithUser]>. Do you have any idea?


Answer:

solution is:

ra.map { (r) -> ReleaseWithUser in
// some condition and logic here
}.flatten(on: request)

Array can be converted to future array with flatten().

Question:

I have the following section of code in a controller, which queries MailGun to send an email, and then should ideally wait for a response before returning in the controller. As configured right now, this should fail as I have intentionally broken my MailGun configuration. But currently the controller is returning a success status, because the .catchMap functionality is not being properly awaited, and I'm not sure how to correctly structure my code so that it is.

return emailTemplate.render(emailData, on: req).map { message -> Future<Response> in
    let deliveryService = try req.make(EmailDeliveryService.self)
    return try deliveryService.send(message, on: req)
}.catchMap { error in
    /// this is not being awaited, and no abort is thrown before the request returns
    throw Abort(.internalServerError, reason: "Error sending email.")
}.transform(to: savedObj)

The function that should be properly awaited, deliverService.send, has the method signature:

func send(_ message: EmailMessage, on container: Container) throws -> Future<Response>

How can I appropriately structure this code to correctly catch an error that is returned by the result of the deliveryService.send method?


Answer:

If your render() method signature is something like this:

func render(…) -> Future<String>

then, I think you need to use flatMap instead of map in:

return emailTemplate.render(emailData, on: req).map // <—

Right now, the catchMap and transform method are receiving a Future<Future<Response>> because map only transform the encapsulated data of the given future which go like this:

Future<String> -map(String -> Future<Response)-> Future<Future<Response>>

Using flatMap, it will flatten the double Future, which is the purpose of this method which lead to:

Future<String> -flatMap(String -> Future<Response)-> Future<Response>

Then, the catchMap will be able to access the error.