How to store files with meta data in LoopBack?

loopback upload file remote method
loopback remote method download file
loopback 3
loopback cluster
loopback aws
loopback 3 example github
loopback book
what is loopback io

What I want to do: Have an html form, with a file input inside. When a file is chosen, the file input should upload the file, and get a file id, so when the form is submitted, the file id is posted with the form and written in the database.

Shorter version: I want to store meta data (id for example) with my files.

Sounds simple, yet I struggle to do that in LoopBack.

There has been a couple conversations ( 1, 2 ) about this topic, and neither seemed to lead to a solution, so I thought this might be a good place to find one once and for all.

The simplest solution would be to use model relations, but LoopBack doesn't support relations with the file storage service. Bump. So we have to go with a persistedmodel named File for example, and override default create, delete so it saves and deletes from the file store model I have - named Storage.

My setup so far:

  • I have a model /api/Storage which is connected to a loopback storage service and is saving file successfully to the local filesystem.
  • I have a PersistedModel connected to Mongo with file meta data: name,size, url and objectId
  • I have a remote hook set up beforecreate so the file can be saved first and then it's url can be injected into File.create()

I'm there, and according to this LoopBack page, I have the ctx which should have the file inside:

File.beforeRemote('create', function(ctx, affectedModelInstance, next) {})`

What's ctx?

ctx.req: Express Request object. ctx.result: Express Response object.

Ok, so now I'm at the Express page, pretty lost, and it sais something about a 'body-parsing middleware' which I have no idea what it might be.

I feel like I'm close to the solution, any help would be appreciated. Is this approach right?

Here's the full solution for storing meta data with files in loopback.

You need a container model

common/models/container.json

{
  "name": "container",
  "base": "Model",
  "idInjection": true,
  "options": {
    "validateUpsert": true
  },
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}

Create the data source for your container in server/datasources.json. For example:

...
"storage": {
    "name": "storage",
    "connector": "loopback-component-storage",
    "provider": "filesystem", 
    "root": "/var/www/storage",
    "maxFileSize": "52428800"
}
...

You'll need to set the data source of this model in server/model-config.json to the loopback-component-storage you have:

...
"container": {
    "dataSource": "storage",
    "public": true
}
...

You'll also need a file model to store the meta data and handle container calls:

common/models/files.json

{
  "name": "files",
  "base": "PersistedModel",
  "idInjection": true,
  "options": {
    "validateUpsert": true
  },
  "properties": {
    "name": {
      "type": "string"
    },
    "type": {
      "type": "string"
    },
    "url": {
      "type": "string",
      "required": true
    }
  },
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}

And now connect files with container:

common/models/files.js

var CONTAINERS_URL = '/api/containers/';
module.exports = function(Files) {

    Files.upload = function (ctx,options,cb) {
        if(!options) options = {};
        ctx.req.params.container = 'common';
        Files.app.models.container.upload(ctx.req,ctx.result,options,function (err,fileObj) {
            if(err) {
                cb(err);
            } else {
                var fileInfo = fileObj.files.file[0];
                Files.create({
                    name: fileInfo.name,
                    type: fileInfo.type,
                    container: fileInfo.container,
                    url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name
                },function (err,obj) {
                    if (err !== null) {
                        cb(err);
                    } else {
                        cb(null, obj);
                    }
                });
            }
        });
    };

    Files.remoteMethod(
        'upload',
        {
            description: 'Uploads a file',
            accepts: [
                { arg: 'ctx', type: 'object', http: { source:'context' } },
                { arg: 'options', type: 'object', http:{ source: 'query'} }
            ],
            returns: {
                arg: 'fileObject', type: 'object', root: true
            },
            http: {verb: 'post'}
        }
    );

};

For expose the files api add to the model-config.json file the files model, remember select your correct datasources:

...
"files": {
    "dataSource": "db",
    "public": true
}
...

Done! You can now call POST /api/files/upload with a file binary data in file form field. You'll get back id, name, type, and url in return.

Storage component, slc loopback:datasource [?] Enter the data-source name: storage [?] Select the This returns metadata about the file, not the file itself. The form  When a file is chosen, the file input should upload the file, and get a file id, so when the form is submitted, the file id is posted with the form and written in the database. Shorter version: I want to store meta data (id for example) with my files. Sounds simple, yet I struggle to do that in LoopBack.

I had the same problem. I solved it by creating my own models to store meta data and my own upload methods.

  1. I created a model File which will store info like name,type,url,userId ( same as yours)

  2. I created my own upload remote method because I was unable to do it with the hooks. Container model is the model which is created by loopback-component-storage.

  3. var fileInfo = fileObj.files.myFile[0]; Here myFile is the fieldname for file upload, so you will have to change it accordingly. If you don't specify any field, then it will come as fileObj.file.null[0]. This code lacks proper error checking, do it before deploying it in production.

     File.uploadFile = function (ctx,options,cb) {
      File.app.models.container.upload(ctx.req,ctx.result,options,function (err,fileObj) {
        if(err) cb(err);
        else{
                // Here myFile is the field name associated with upload. You should change it to something else if you
                var fileInfo = fileObj.files.myFile[0];
                File.create({
                  name: fileInfo.name,
                  type: fileInfo.type,
                  container: fileInfo.container,
                  userId: ctx.req.accessToken.userId,
                  url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name // This is a hack for creating links
                },function (err,obj) {
                  if(err){
                    console.log('Error in uploading' + err);
                    cb(err);
                  }
                  else{
                    cb(null,obj);
                  }
                });
              }
            });
    };
    
    File.remoteMethod(
      'uploadFile',
      {
        description: 'Uploads a file',
        accepts: [
        { arg: 'ctx', type: 'object', http: { source:'context' } },
        { arg: 'options', type 'object', http:{ source: 'query'} }
        ],
        returns: {
          arg: 'fileObject', type: 'object', root: true
        },
        http: {verb: 'post'}
      }
    
    );
    

Working with File Storage and LoopBack, Shorter version: I want to store meta data (id for example) with my files. Sounds simple, yet I struggle to do that in LoopBack. There has been a  You use the storage component like any other LoopBack data source such as a database. Like other data sources, it supports create, read, update, and delete (CRUD) operations with exactly the same LoopBack and REST APIs. This component does not yet provide metadata management “out of the box”.

For those who are looking for an answer to the question "how to check file format before uploading a file".

Actual in this case we can use optional param allowedContentTypes.

In directory boot use example code:

module.exports = function(server) {
    server.dataSources.filestorage.connector.allowedContentTypes = ["image/jpg", "image/jpeg", "image/png"];
}

I hope it will help someone.

Download file and it's metadata with one call · Issue #124 , I know that loopback-storage-component is not yet provide metadata saving while upload a file. So someone solved this problem by creating a  When a file is chosen, the file input should upload the file, and get a file id, so when the form is submitted, the file id is posted with the form and written in the database. Shorter version: I want to store meta data (id for example) with my files. Sounds simple, yet I struggle to do that in LoopBack.

Depending on your scenario, it may be worth looking at utilising signatures or similar allowing direct uploads to Amazon S3, TransloadIT (for image processing) or similar services.

Our first decision with this concept was that, as we are using GraphQL, we wanted to avoid multipart form uploads via GraphQL which in turn would need to transfer to our Loopback services behind it. Additionally we wanted to keep these servers efficient without potentially tying up resources with (large) uploads and associated file validation and processing.

Your workflow might look something like this:

  1. Create database record
  2. Return record ID and file upload signature data (includes S3 bucket or TransloadIT endpoint, plus any auth tokens)
  3. Client uploads to endpoint

For cases where doing things like banner or avatar uploads, step 1 already exists so we skip that step.

Additionally you can then add SNS or SQS notifications to your S3 buckets to confirm in your database that the relevant object now has a file attached - effectively Step 4.

This is a multi-step process but can work well removing the need to handle file uploads within your core API. So far this is working well from our initial implementation (early days in this project) for things like user avatars and attaching PDFs to a record.

Example references:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html

https://transloadit.com/docs/#authentication

Multipart Data Upload · Issue #113 · strongloop/loopback , Am using ngFileUpload file upload but passing any data in the .com/questions/​28885282/how-to-store-files-with-meta-data-in-loopback  Overview. The LoopBack storage component makes it easy to upload and download files to cloud storage providers and the local (server) file system. It has Node.js and REST APIs for managing binary content in cloud providers, including:

For anyone else having that problem with loopback 3 and Postman that on POST, the connection hangs (or returns ERR_EMPTY_RESPONSE) (seen in some comments here)... The problem in this scenario is, that Postman uses as Content-Type "application/x-www-form-urlencoded"!

Please remove that header and add "Accept" = "multipart/form-data". I've already filed a bug at loopback for this behavior

Storing files in databases with LoopBack - Jérémie Drouet, In MongoDB, it's called GridFS and it's a bit more interesting because it allows to add metadata to the stored file. With this, there's no need to  Shorter version: I want to store meta data (id for example) with my files. Sounds simple, yet I struggle to do that in LoopBack. There has been a couple conversations ( 1 , 2 ) about this topic, and neither seemed to lead to a solution, so I thought this might be a good place to find one once and for all.

Loopback and Image Uploads, a File model to store the image data, attached to "storage" datasource. - a FileMetadata to store image metadata, attached to your MongoDB datasource. Creating a Loopback File System (LOFS) A LOFS file system is a virtual file system that provides an alternate path to an existing file system. When other file systems are mounted onto an LOFS file system, the original file system does not change. For more information, see the lofs(7FS).

Use the Device Mapper storage driver, This is indicated by the fact that the Data loop file and a Metadata loop file are on files under /var/lib/docker/  I have added meta data by referring How to store files with meta data in LoopBack? Now I have to check if the filetype is in csv, before uploading it to the server. Right now, I delete the uploaded file if it is not valid. Is there a better way to solve this?

Orchestrating Docker, The devicemapper driver creates a thin pool of storage blocks based on two block block devices are created by mounting sparse files as loopback devices. It tracks the location of the written and empty blocks through the file's metadata. Dismiss Join GitHub today. GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.

Comments
  • I can get data for File.beforeRemote('upload', function(ctx, modelInstance, next){ console.log(ctx.req); next(); }); , however I can't see any file related information in the ctx object, and the modelInstance is undefined too... Worth noting that my File here is the model with the storage service datasource.
  • Thanks RYFN for taking a look into this. For the sake of consistency, I'll stick with my naming 'File' for the file meta data and storageId, and 'Storage' for the file model bound to storage service.
  • I can easily do a remote hook to Storage.upload, and get file meta data like name, size, etc., and call File.create() from the hook, but this is not the best solution. File being a persistentModel can be set to be related to User.profileimage for example, and if a user posts a form with the image inside, it would be handled well by Loopback. So I'm still looking for a solution with a hook to File and not Storage models.
  • how do you get the file meta data out of the .upload hook? would you be able to show an example?
  • Storage.afterRemote('upload',function(ctx, modelInstance, next){ console.log('create file',modelInstance.result.files.file); next(); });
  • File.app is not defined :(
  • make sure you take File as the attribute for your hook module.exports = function(File) {...}
  • Hey I get a response saying [Error: Request aborted]. I have done exactly what u have done. Any leads would be awsm. thanks.
  • @NileshG, sure in File.create({ name: fileInfo.name, type: fileInfo.type, container: fileInfo.container, url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name },..... you can set name to whatever.
  • the above config give me .('Cannot override built-in "{{file}}" type.'));
  • Awsome, your answer guided me to the solutions I was looking for. I'll accept yours, and will upload the full solution with model defs for the record.
  • great! for any one like me who got ( can't find upload of undefined ) make sure your container model is named container small case! or change it!