What is better datastructure to store user profiles in redis?

redis data model
how to store data in redis
redis tutorial
redis commands
redis documentation

I want to store user's profiles in redis, as I have to frequently read multiple user's profiles.. there are two options I see at present:

Option 1: - store separate hash key per user's profile

  • [hash] - u1 profile {id: u1, name:user1, email:user1@domain.com, photo:url}
  • [hash] - u2 profile {id: u2, name:user2, email:user2@domain.com, photo:url}
  • where for every user's id is hash key and profile field and values JSON-serialized profile objects. (OR instead of json user field-value pairs)

Option 2: - use single hash key to store all users profile

  • [hash] - users-profile u1 {id: u1, name:user1, email:user1@domain.com, photo:url}
  • [hash] - users-profile u2 {id:u2, name:user2, email:user2@domain.com, photo:url}
  • where in users-profile hash key, user's ids field and values JSON-serialized profile objects.

Please tell me which option is best considering following:

  1. performance
  2. memory utilization
  3. read multiple user's profile - for batch processing I should able to read 1-100, 101-200 user's profile at time
  4. larger dataset - what if there are millions users profile

As Sergio Tulentsev pointed out, its not good to store all the user's data (especially if the dataset is huge) inside one single hash by any means.

Storing the users data as individual keys is also not preferred if your looking for memory optimization as pointed out in this blog post

Reading the user's data using pagination mechanism demands one to use a database rather than a simple caching system like redis. Hence it's recommended to use a NoSQL database such as mongoDB for this.

But reading from the database each time is a costly operation especially if you're reading a lot of records.

Hence the best solution would be to cache the most active user's data in redis to eliminate the database fetch overhead.

I recommend you looking into walrus .

It basically follows the following pattern:

def function_name(param1, param2, ...., param_n):
    # perform database fetch
    # return user data

This ensures that the frequently accessed or requested user data is in redis and the function automatically returns the value from redis than making the database call. Also the key is expired if not accessed for a long time.

You set it up as follows:

from walrus import *
db = Database(host='localhost', port=6379, db=0)

where host can take the domain name of the redis cluster running remotely.

Hope this helps.

Data Structures, What Redis data structure can be used to store key value pairs? As shown in table 1.1, Redis allows us to store keys that map to any one of five different data structure types; STRINGs, LISTs, SETs, HASHes, and ZSETs. Each of the five different structures have some shared commands (DEL, TYPE, RENAME, and others), as well as some commands that can only be used by one or two of the structures.

Redis: What and Why?, one or more fields in the hash as a counter. Redis as an in-memory data store with high availability and persistence is a popular choice among application developers to store and manage session data for internet-scale applications. Redis provides the sub-millisecond latency, scale, and resiliency required to manage session data such as user profiles, credentials, session state, and user

FAQ – Redis, , used as a database, cache and message broker. It supports various data structures such as Strings, Hashes, Lists, Sets etc. User Profiles: Many web applications use Redis Hashes for their user profiles, as they can use a single hash for all the user fields, such as name, surname, email, password, etc. User Posts: Social platforms like Instagram leverage Redis Hashes to map all the archived user photos or posts back to a single user.

When should I use Redis as my primary data store?, Instead, Redis stores data in data structures which makes it very flexible to use. If there are more urgent jobs you require to be executed, these can be User Profiles: Many web applications use Redis Hashes for their user  In Redis you won't store your data in a unique data structure. You have to create multiple data structure, each one being identified by a key. Use a convention to name yours keys: by example site:<CITY>:buildings will be a set that contains the list of building ids for a given site. Then define hashes to store each building description.

Top Redis Use Cases by Core Data Structure Types, This book covers the use of Redis, an in-memory database/data structure server. We'll store user information inside of Redis as a HASH , similar to how we stored Listing 8.1How to create a new user profile HASH ed passwords, email addresses, and more are stored at other keys, or in a different database entirely. Caching is the process of storing some data in Cache. Cache is a temporary storage component area where the data is stored so that in future, Data can be served faster. According to Redis homepage, Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

8.1.1 User information, Join us for RedisConf 2020 Takeaway, starting May 12 Learn More ×. × This book covers the use of Redis, an in-memory database/data structure server. that a user is following, we'll also store user IDs and timestamps in ZSET s as well​, with need to be updated, as well as counts in the two user profile HASH es. HLLs in Redis, while technically a different data structure, are encoded as a Redis string, so you can call GET to serialize a HLL, and SET to deserialize it back to the server. Conceptually the HLL API is like using Sets to do the same task.