A Resource Manager for Game Assets

Published September 30, 2014 by Davide Cleopesce, posted by TheItalianJob71
Do you see issues with this article? Let us know.
Advertisement

Preamble

This article has been written based on my personal experience, if you think you are offended in some ways because of my personal opinions about programming and / or my coding style, please stop reading now. I am not a guru programmer and I don't want to impose anything on anyone.

Introduction

During the development of my engine(s), I have always had the need for better handling of game assets, even if in the past it was sufficient to hard code the needed resources at startup. Now with the added speed of modern computers it is possible to create something more advanced, dynamic and general-purpose. The most important points I needed for a resource manager were:

  • reference counted resources.
  • If a resource is already present in the database, expose it as raw pointer, else load it.
  • know automatically when to free the resource.
  • fast access using string to retrieve resources
  • clean everything on exit without manually deleting the resources.
  • mapping resource using a referenced structure.

At this point it looks like I am going to reinvent the wheel, given the assumption that smart pointers are now included with the new C++ standard. My personal problem with the smart pointers is that it was very difficult to create the kind of data structure I wanted. My idea was to create an unordered map of shared pointers and then hand out a weak pointer when the resource was asked for. The problem with this data organization is that the weak pointer basically 'observes' the resource but doesn't hold it. On the contrary using an unordered map of weak pointers and handing out a shared pointer created different problems, which could be resolved using a custom deleter and other indirect strategies. One of this was that if I loaded the resource for the first time, its count was set to 1 and when the same resource was asked for its count was increased to 2. So I still had the problem to ignore the first reference, which was again solvable using a custom deleter and tracking how many resources where effectively still left. Further more, shared pointers are thread safe and according to the standard they are synced even if there is no strict necessity. I am not saying that smart pointer are useless, they are used quite extensively, but my approach needed something different; I needed to have total control of reference counting. Basically what I needed was a wrapper class to store the pointer to the loaded resource, and its reference counting. Note that the code I copied from my engine uses some other utility functions - they are not necessary, because they handle error messaging and string 'normalisation' (eliminating white spacing and lowering the string down), so you can easily ignore those functions and substitute them with yours. I have also removed all debugging printing during the execution, to keep things clearer. Let's have a look at the base class for resources.

class CResourceBase 
{ 
    template < class T > friend class CResourceManager; 

private: 
    int References; 
    void IncReferences() { References++; } 
    void DecReferences() { References--; } 
    
protected: 
    // copy constructor and = operator are kept private 
    CResourceBase(const CResourceBase& object) { } 
    CResourceBase& operator=(const CResourceBase& object) { return *this; } 
    
    // resource filename 
    std::string ResourceFileName; 
    
public: 
    const std::string &GetResourceFileName() const { return ResourceFileName; } 
    const int GetReferencesCount() const { return References; } 
    
    //////////////////////////////////////////////// 
    // ctor / dtor 
    CResourceBase( const std::string& resourcefilename ,void *args ) 
    { 
        // exit with an error if filename is empty 
        if ( resourcefilename.empty() ) CMessage::Error("Empty filename not allowed"); 
        
        // init data members 
        References = 0; 
        ResourceFileName=CStringFormatter::TrimAndLower( resourcefilename ); 
    } 
    
    virtual ~CResourceBase() { } 
}; 

The class is self-explanatory - the interesting point here is the constructor, which needs a resource filename, including the full path of your resource and a void pointer to an argument class in case you wanted to load the resource with some initial default parameters. The args pointer comes in handy when you want to instantiate assets during runtime and don't want to load them. There are some cases where this is useful, for every other case the constructor will serve our purposes well. All of our assets will inherit from this class. There is, obviously, the reference counter and some functions for accessing it.

The Resource Manager

The resource manager basically is a wrapper for an unordered map. It uses the string as a key and maps it to a raw pointer. I have decided to use an unordered map because I don't need a sorted arrangement of assets, I really do care at retrieving them in the fastest possible way, in fact accessing a map is O(log N), while for an unordered map is O(N). In addition just because the unordered map is constant speed (O(1)) doesn't mean that it's faster than a map (of order log(N)). Anyway, in my test cases the N value wasn't so huge, so the unordered map has always been faster than map, thus I decided to use this particular data structure as the base of my resource manager. The most important functions in the resource map are Load and Unload. The Load function tries to retrieve the assets from the unordered map. If the asset is present, its reference count is increased and the wrapped pointer is returned. If it's not found in the database, the function creates a new asset class, increases its reference count, stores it in the map and returns its wrapped pointer. Note that its the programmer's responsibility to create an asset class with a proper constructor. The base class, inherited from CResourceBase class, must provide a string containing the full path from where the asset needs to be loaded and an argument class if any - this will be clearer when the example is provided. The Unload function does exactly the opposite: looks for the requested asset given its file name, if the resource is found, its reference counter is decreased, and if it reaches zero the associated memory is released. Since I think that a good programmer understands better 1000 lines of code rather than 1000 lines of words, here you have the entire resource manager:

template < class T > class CResourceManager 
{ 
private: 
    // data members 
    std::unordered_map< std::string, T* > Map; 
    std::string Name; 
    
    // copy constructor and = operator are kept private 
    CResourceManager(const CResourceManager&) { }; 
    CResourceManager &operator = (const CResourceManager& ) { return *this; } 
    
    // force removal for each node 
    void ReleaseAll() 
    { 
        std::unordered_map< std::string, T* >::iterator it=Map.begin(); 
        
        while ( it!=Map.end() ) { delete (*it).second; it=Map.erase( it ); } 
    } 
    
public: 
    /////////////////////////////////////////////////// 
    // add an asset to the database 
    T *Load( const std::string &filename, void *args ) 
    { 
        // check if filename is not empty 
        if ( filename.empty() ) 
            CMessage::Error("filename cannot be null"); 
            
        // normalize it 
        std::string FileName=CStringFormatter::TrimAndLower( filename ); 
        
        // looks in the map to see if the 
        // resource is already loaded 
        std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); 
        
        if (it != Map.end()) 
        { 
            (*it).second->IncReferences(); 
            return (*it).second; 
        } 
        
        // if we get here the resource must be loaded 
        // allocate new resource using the raii paradigm 
        // you must supply the class with a proper constructor 
        // see header for details 
        T *resource= new T( FileName, args ); 
        
        // increase references , this sets the references count to 1 
        resource->IncReferences(); 
        
        // insert into the map 
        Map.insert( std::pair< std::string, T* > ( FileName, resource ) );
        
        return resource; 
    } 
    
    /////////////////////////////////////////////////////////// 
    // deleting an item 
    bool Unload ( const std::string &filename ) 
    { 
        // check if filename is not empty 
        if ( filename.empty() ) 
            CMessage::Error("filename cannot be null"); 
            
            // normalize it 
            std::string FileName=CStringFormatter::TrimAndLower( filename ); 
            
            // find the item to delete 
            std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); 
            
            if (it != Map.end()) 
            { 
                // decrease references 
                (*it).second->DecReferences(); 
                
                // if item has 0 references, means 
                // the item isn't more used so , 
                // delete from main database 
                if ( (*it).second->GetReferencesCount()==0 ) 
                { 
                    // call the destructor 
                    delete( (*it).second ); 
                    Map.erase( it ); 
                } 
                
                return true; 
            } 
            
            CMessage::Error("cannot find %s\n",FileName.c_str()); 
            
            return false; 
        } 
        
        ////////////////////////////////////////////////////////////////////// 
        // initialise 
        void Initialise( const std::string &name ) 
        { 
            // check if name is not empty 
            if ( name.empty() ) 
                CMessage::Error("Null name is not allowed"); 
                
            // normalize it 
            Name=CStringFormatter::TrimAndLower( name ); 
        } 
        
        //////////////////////////////////////////////// 
        // get name for database 
        const std::string &GetName() const { return Name; } 
        const int Size() const { return Map.size(); } 
        
        /////////////////////////////////////////////// 
        // ctor / dtor 
        CResourceManager() { } 
        ~CResourceManager() { ReleaseAll(); 
    } 
};

Mapping Resources

The resource manager presented here is fully functional of its own, but we want to be able to use assets inside a game object represented by a class. Think about a 3D object, which is made of different 3D meshes, combined together in a sort of hierarchial structure, like a simple robot arm, makes the idea clearer. The object is composed of simple building blocks, like cubes and cylinders. we want to reuse every object as much as possible and also we want to access them quickly, in case we want to rotate a single joint. The engine must fetch the object quickly, without any brute force approach, also we want a name for the asset so we can address it using human readable names, which are easier to remember and to organize. The idea is to write a resource mapper which uses another unordered map using strings as keys and addresses from the resource database as the mapped value. We need also to specify if we want to allow the asset to be present multiple times or not. The reason behind this is simple - think again at the 3D robot arm. We need to use multiple times a cube for example, but if we use the same resource mapper for a shader, we need to keep each of the shaders only once. Everything will become clearer as the code for the mapper unfolds further ahead.

template < class T > class CResourceMap 
{ 
private: 
    ///////////////////////////////////////////////////////// 
    // find in all the map the value requested 
    bool IsValNonUnique( const std::string &filename ) 
    { 
        // if duplicates are allowed , then return alwasy true 
        if ( Duplicates ) return true; 

        // else , check if element by value is already present 
        // if it is found, then rturn treu, else exit with false 
        std::unordered_map< std::string, T* >::iterator it= Map.begin(); 
        while( it != Map.end() ) 
        { 
            if ( ( it->second->GetResourceFileName() == filename ) ) 
                return false; 
                
            ++it; 
        } 
        
        return true; 
    } 
    
    ////////////////////////////////////////////////////////////////////////////// 
    // private data 
    std::string Name; 
    
    // name for this resource mapper 
    int Verbose; 
    
    // flag for debugging messages 
    int Duplicates; 
    
    // allows or disallwos duplicated filenames for resources 
    CResourceManager *ResourceManager; 
    
    // attached resource manager 
    std::unordered_map< std::string, T* > Map; 
    
    // resource mapper 
    // copy constructor and = operator are kept 
    private CResourceMap(const CResourceMap&) { }; 

    CResourceMap &operator = (const CResourceMap& ) { return *this; } 
    
public: 
    ////////////////////////////////////////////////////////////////////////////////////// 
    // adds a new element 
    T *Add( const std::string &resourcename,const std::string &filename,void *args=0 ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (5)" ); 
            
        if ( filename.empty() ) 
            CMessage::Error("%s : filename cannot be null",Name.c_str()); 
        
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        
        // looks in the hashmap to see if the 
        // resource is already loaded 
        std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); 
        
        if ( it==Map.end() ) 
        { 
            std::string FileName=CStringFormatter::TrimAndLower( filename ); 
            
            // if duplicates flag is set to true , duplicated mapped values 
            // are allowed, if duplicates flas is set to false, duplicates won't be allowed 
            if ( IsValNonUnique( FileName ) ) 
            { 
                T *resource=ResourceManager->Load( FileName,args ); 
                
                // allocate new resource using the raii paradigm 
                Map.insert( std::pair< std::string, T* > ( ResourceName, resource ) ); 
                return resource; 
            } else { 
                // if we get here and duplicates flag is set to false 
                // the filename id duplicated 
                CMessage::Error("Filename name %s must be unique\n",FileName.c_str() ); 
            } 
        } 
        
        // if we get here means that resource name is duplicated 
        CMessage::Error("Resource name %s must be unique\n",ResourceName.c_str() ); 
        return nullptr; 
    } 
    
    ///////////////////////////////////////////////////////// 
    // delete element using resourcename 
    bool Remove( const std::string &resourcename ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (4)"); 
            
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        
        if ( Verbose ) 
            CMessage::Trace("%-64s: Removal proposal for : %s\n",Name.c_str(),ResourceName.c_str() ); 
            
        // do we have this item ? 
        std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); 
        
        // yes, delete element, since it is a reference counted pointer, 
        // the reference count will be decreased 
        if ( it != Map.end() ) 
        { 
            // save resource name 
            std::string filename=(*it).second->GetResourceFileName(); 
            
            // erase from this map 
            Map.erase ( it ); 
            
            // check if it is unique and erase it eventually 
            ResourceManager->Unload( filename ); 
            return true; 
        } 
        
        // if we get here , node couldn't be found 
        // so , exit with an error 
        CMessage::Error("%s : couldn't delete %s\n",Name.c_str(), ResourceName.c_str() ); 
        return false; 
    } 
    
    ////////////////////////////////////////////////////////// 
    // clear all elements from map 
    void Clear() 
    { 
        std::unordered_map< std::string, T* >::iterator it=Map.begin(); 
    
        // walk trhough all the map 
        while ( it!=Map.end() ) 
        { 
            // save resource name 
            std::string filename=(*it).second->GetResourceFileName(); 
            
            // clear from this map 
            it=Map.erase ( it ); 
            
            // check if it is unique and erase it eventually 
            ResourceManager->Unload( filename ); 
        } 
    } 

    ////////////////////////////////////////////////////////// 
    // dummps database content to a 
    string std::string Dump() 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (3)");

        std::string str=CStringFormatter::Format("\nDumping database %s\n\n",Name.c_str() ); 
        
        for ( std::unordered_map< std::string, T* >::iterator it = Map.begin(); it != Map.end(); ++it ) 
        { 
            str+=CStringFormatter::Format("resourcename : %s , %s\n", (*it).first.c_str(), (*it).second->GetResourceFileName().c_str() ); 
        } 
        
        return str; 
    } 

    ///////////////////////////////////////////////////////// 
    // getters 
    ///////////////////////////////////////////////////////// 
    // gets arrays name 
    const std::string &GetName() const { return Name; } 
    const int Size() const { return Map.size(); } 

    ////////////////////////////////////////////////////////// 
    // gets const reference to resource manager const 
    CResourceManager *GetResourceManager() { return ResourceManager; } 

    ///////////////////////////////////////////////////////// 
    // gets element using resourcename, you should use this 
    // as a debug feature or to get shared pointer and later 
    // use it , using it in a section where performance is 
    // needed might slow down things a bit 
    T *Get( const std::string &resourcename ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (2)"); 
            
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        std::unordered_map< std::string, T* >::iterator it; 
        
        if ( Verbose ) 
        { 
            CMessage::Trace("%-64s: %s\n",Name.c_str(),CStringFormatter::Format("Looking for %s",ResourceName.c_str() ).c_str()); 
        } 
        
        // do we have this item ? 
        it = Map.find( ResourceName ); 
        
        // yes, return pointer to element 
        if ( it != Map.end() ) 
            return it->second; 
        
        // if we get here , node couldn't be found thus , exit with a throw 
        CMessage::Error("%s : couldn't find %s",Name.c_str(), ResourceName.c_str() ); 
        
        // this point is never reached in case of failure 
        return nullptr; 
    } 

    ///////////////////////////////////////////////////////// 
    // setters 
    void AllowDuplicates() { Duplicates=true; } 
    void DisallowDuplicates() { Duplicates=false; } 
    void SetVerbose() { Verbose=true; } 
    void SetQuiet() { Verbose=false; } 

    //////////////////////////////////////////////////////////// 
    // initialise resource mapper 
    void Initialise( const std::string &name, CResourceManager *resourcemanager, bool verbose,bool duplicates ) 
    { 
        if ( resourcemanager==NULL ) 
            CMessage::Error("DataBase cannot be NULL 1"); 
            
        if ( name.empty() ) 
            CMessage::Error("Array name cannot be null"); 
            
        Name=CStringFormatter::TrimAndLower( name ); 
        
        // normalized name 
        string ResourceManager=resourcemanager; 
        
        // copy manager pointer 
        // setting up verbose or quiet mode 
        Verbose=verbose; 
        
        // setting up allowing or disallowing duplicates 
        Duplicates=duplicates; 
        
        // emit debug info 
        if ( Verbose ) 
        { 
            if ( Duplicates ) 
                CMessage::Trace("%-64s: Allows duplicates\n",Name.c_str() ); 
            else if ( !Duplicates ) 
                CMessage::Trace("%-64s: Disallows duplicates\n",Name.c_str() ); 
        } 
    } 

    ///////////////////////////////////////////////////////// 
    // ctor / dtor 
    CResourceMap() 
    { 
        Verbose=-1; // undetermined state 
        Duplicates=-1; // undetermined state 
        
        ResourceManager=NULL; 
        // no resource manager assigned 
    } 

    ~CResourceMap() 
    { 
        if ( Verbose ) 
            CMessage::Trace("%-64s: Releasing\n",Name.c_str() ); 
        
        Clear(); 
            
            // remove elements if unique 
    } 
}; 

Basically, the class is a wrapper for the resource database operations. Among the private data, as you can see, its present an unordered map where the first key is a string and the mapped value is the pointer directly mapped from the resource database. Let's have a look at the function members now. The Add function performs many tasks. First, checks if the name for the asset is already present, since duplicated names for the assets are not allowed. If name is not present, it performs the attempt to upload the assets from the resource database, then it checks if the filename is unique and if the duplicates flag is not set to true. Here I have used a brute force approach, the reason behind it is that if I wanted to have a sort of bidirectional mapping, I should have used a more complex data structure, but I wanted to keep things simple and stupid. At this point the resource database uploads the asset, and if it's present, it hands back immediately the address for the required resource. If not it loads it, making the process completely transparent for the resource mapper and it gets stored in the unsorted map data structure. Note again, that all the error checking are just wrappers for a throw, you may want to replace with your error checking code, without compromising the Add function itself. The Remove function is a little bit more interesting, basically the safety checks are the same used in Add, the resource is erased from the map, the resource database removal function is invoked, but the resource database doesn't destroy it if it is still shared in some other places. By 'some other places' I mean that the asset may be still be present in the same resource mapper or in another resource mapper instantiated somewhere else in your game. This will be clearer with a practical example further ahead. The Clear function basically performs the erasure of the entire resource map, using the same counted reference mechanism from the resource database. The Get function retrieves the named resource by specifing its resource name and gives back the resource pointer. The Initialise function attaches the resource mapper to the resource database.

Example of Usage

First of all, we need a base class, which could be a game object. Let's call it foo just for example

class CFoo : public vml::CResourceBase 
{ 
    //////////////////////////////////////////////////// 
    // copy constructor is private 
    // no copies allowed since classes 
    // are referenced 
    
    CFoo( const CFoo &foo ) : CResourceBase ( foo ) { } 
    
    //////////////////////////////////////////////////// 
    // overload operator is private, 
    // no copies allowed since classes 
    // are referenced 
    CFoo &operator =( CFoo &foo ) 
    { 
        if ( this==&foo ) 
            return *this; 

        return *this; 
    } 
    
public: 
    //////////////////////////////////////////////// 
    // ctor / dtor 
    // this constructor must be present 
    CFoo(const std::string &resourcefilename, void *args ) : CResourceBase( resourcefilename,args ) { } 
    
    // regular base constructor and destructor 
    Cfoo() {} ~CFoo() { } 
}; 

Now we can instantiate our resource database and resource mappers.

CResourceManager rm; 
CResourceMap mymap1; 
CResourceMap mymap2;

I have createed a resource manager and two resource mappers here.

// create a resource database 
rm.Initialise("FooDatabase", vml::CResourceManager::VERBOSE ); 

// attach this database to the resource mappers, bot of them allowd duplicates 
mymap1.Initialise( "foolist1",&rm, true,true ); 
mymap2.Initialise( "foolist2",&rm, true,true ); 

// populate first resoruce mapper 
// the '0' argument means that the resource 'a' , whose filename is foo1.txt 
// doesn't take any additional values at construction time 
mymap1.Add( "a","foo1.txt",0 ); 
mymap1.Add( "b","foo1.txt",0 ); 
mymap1.Add( "c","foo2.txt",0 ); 
mymap1.Add( "d","foo2.txt",0 ); 
mymap1.Add( "e","foo1.txt",0 ); 
mymap1.Add( "f","foo1.txt",0 ); 
mymap1.Add( "g","foo3.txt",0 ); 

// populate second resource mapper 
mymap2.Add( "a","foo3.txt",0 ); 
mymap2.Add( "b","foo1.txt",0 ); 
mymap2.Add( "c","foo3.txt",0 ); 
mymap2.Add( "d","foo1.txt",0 ); 
mymap2.Add( "e","foo2.txt",0 ); 
mymap2.Add( "f","foo1.txt",0 ); 
mymap2.Add( "g","foo2.txt",0 ); 

// dump content into a stl string which can be printed as you like 
std::string text=rm.Dump();

Running this example and printing the text content gives:

Dumping database foodatabase 
Filename : foo1.txt , references : 7 
Filename : foo2.txt , references : 4 
Filename : foo3.txt , references : 3

This concludes the article. I hope it will be useful for you, thanks for reading.

Cancel Save
1 Likes 25 Comments

Comments

EDI
EDI

I will chime-in and say I am happy at least to see it was not implemented as a Singleton.

September 30, 2014 08:29 PM
Ravyne

Singleton is one of the biggest problems with 'manager' classes, so the author has done well to avoid it. I'll raise your chime-in to say that relying on a resource base class is intrusive, heavy, and brittle, and that the code could be simplified by using C++11 smart pointers to handle lifetime management and ownership.

I had a pretty nice pre-C++11 resource manager that used boost's shared pointer, that I had always meant to write an article on. Maybe I should revisit that code and update to make use of useful C++11/C++14 features. I can think of a handful that would make for real slick interface/usage.

October 01, 2014 02:11 AM
majo33

I think you can use std::shared_ptr for this. Use use_count() method to check number of references. When the use_count is 1 then your resource is referenced only by the resource manager.

October 01, 2014 06:57 AM
NightCreature83

Sorry but why is the resource manager a Friend of ResourceBase, there is no need for friend access.

October 01, 2014 07:20 AM
TheItalianJob71

Nightcreature, the resource manager uses the IncReferences and DecReferences, which are private members, Majo33, about the use_count, that is the problem i wanted to avoid , using the shared pointer i were in the condition that no resource mapper had the 'ownership' for that asset , but it was still loaded in memory. I tried to overcome the problem, but the code started to look unnecessary complicated.

About the c++11 features , maybe will rewrite the code to use these new features one day.

October 01, 2014 08:11 AM
NightCreature83

When you start using features like friend it generally means you designed your interface wrong and should rethink your strategy of doing this.

October 01, 2014 12:05 PM
the incredible smoker

I,m using seperate level & global data myself.

Global : used in each level, will be cleaned on exit game.

Level : for only 1 level, will be cleaned after exit level.

greets

October 01, 2014 02:56 PM
EDI
EDI

...I also hate to be that guy but the article image uses copywritten artwork from Hyperbole and a Half.

Seems to have been fixed.

October 01, 2014 06:36 PM
Ravyne

>> using the shared pointer i were in the condition that no resource mapper had the 'ownership' for that asset , but it was still loaded in memory.

But that's the behavior you probably want. If someone other than any of the managers holds a reference count (e.g. a shared_ptr), then it should be because they have an interest in the lifetime of the object it points to -- thus, the object *should* remain in memory because, for example, the code that holds the reference might try to render that resource. It shouldn't die just because none of the managers know about it any more.

You have two classes of entities that have an interest in the lifetime of the object -- you have users of the resource that need the thing to be alive as long as they need, regardless of caching; and you have the manager of the resource who's sole interest is caching to share resources, eliminate redundancy, and (optionally) to extend the lifetime of a resource that currently has no users (e.g. keep the thing alive while there's no urgent need to release it, because you think the thing might be needed soon). So, both users and the manager are owners.

The only minor wrinkle is that you may deem it undesirable for there to be multiple copies of an asset in memory, such as when the manager forgets it but it's still being used by somebody, and then another somebody needs that asset later, causing a new, distinct copy to be loaded through a manager. This is sub-optimal, perhaps, but correct. Any pathological, worst-case program behavior that might result from this property should be considered a logic error on the part of client code, although there are ways even for the manager to release its ownership interest in the asset while still being able to find it again -- e.g. a 'victim cache'.

October 01, 2014 06:41 PM
Servant of the Lord

I think you can use std::shared_ptr for this. Use use_count() method to check number of references. When the use_count is 1 then your resource is referenced only by the resource manager.


Alternatively you can have your resource manager store std::weak_ptrs, and call std::weak_ptr::lock() or check std::weak_ptr::expired(). If weak_ptr failed to lock or was expired, then it means it was de-allocated because of zero references (the resource manager's weak_ptr not counting as a reference), so you can then re-load it.

Basically:


typedef TextureKey; //ID, filepath, whatever.
std::unordered_map<TextureKey, std::weak_ptr<Texture>> textures;

std::shared_ptr<Texture> GetTexture(TextureKey key)
{
     //Get the pointer to the texture, automatically creating a null pointer if it doesn't yet exist.
     std::weak_ptr<Texture> &texturePtr = this->textures[key]; 
    
     //If the texture is currently loaded...
     std::shared_ptr<Texture> loadedTexture = texturePtr.lock();
     if(loadedTexture)
     {
          //...return it.
          return loadedTexture;
     }
     
     //Otherwise, load the texture.
     loadedTexture = std::make_shared<Texture>(GetCreationDetails(key));

     //Set the map's pointer to the new texture.
     texturePtr = loadedTexture;

     //Return the pointer.
     return loadedTexture;
}

It automatically frees the resource if there are no longer any living references to it (apart from the manager's reference), and loads it again the next time it is requested.

This could actually even be wrapped up as a single generic template function rather than a templated class or class hierarchy.

October 01, 2014 06:52 PM
Ravyne

One minor wrinkle with with std::weak_ptr referring to a std::shared_ptr made with std::make_shared is that make_shared attempts to optimize spacial locality by combining the object with the control block (the ref-counts and such) for objects up to a certain size (I think a cache-line in total, so 64 bytes minus the size of the control-block itself on most platforms). std::weak_ptr normally would not preserve the object allocation, but it does preserve the control block allocation, thus when they are combined, std::weak_ptr can actually cause the object (albeit a small one) to remain in memory even after no std::shared_ptr holds a reference count. You might or might not consider this to be a problem, but its definitely something to be aware of.

[Note -- leaving this for context, but on closer examination it appears to be incorrect]

If the small object represents a costlier resource (e.g. the small object owns a sizable allocation, device handle, etc) this can become troublesome.

You can prevent this from happening by not using std::make_shared, but you'll give up the locality optimization and pay the normal pointer-hop to dereference the object each time you access it, and likely pay a cache hit.

October 01, 2014 07:40 PM
Servant of the Lord

That's very interesting. You're saying that it doesn't just keep the raw bytes of the 'small object' in memory (which is fine), but it also never calls the destructor on it, until even after all the weak_ptrs go out of scope (which is terrible)?

That's pretty bad. I can't replicate that negative behavior with GCC 4.8.1 - here's my test code. Maybe that only occurs with plain-old-data (not sure how to test for that)? I would imagine they'd call placement new (during construction) and just directly call the destructor when the shared references equal zero.

Here's a better example, just in-case std::shared_ptr::reset() has different behavior.

October 01, 2014 08:36 PM
TheItalianJob71

Question to nightcreature83, I wanted to keep the reference increment and decrement functions private but accessible from the resource manager, preventing the user to artificially modify the reference counting.

Do you know of a better way to achieve this without frinedship?? just curious.

October 01, 2014 09:55 PM
Ravyne

Servant -- Actually, I revise my statement. I'm not certain but you're probably right that the destructor is at least called. Only the "small object" portion would remain in that case. Far less troublesome, but still irksome.

ItalianJob -- You'll notice that shared_ptr doesn't have exposed increment and decrement operations at all -- its all implicitly managed during construction/deconstruction, copy, move, and other special operations like reset(). Through the exposed interface, you can look at the refcount but you can't touch. This is an example of 'maintaining class invariants'. This is the essential function of shared_ptr -- the only other thing it does is hold a pointer to some object and provide a means to hold and then call a special deallocation function if needed.

There's no good way to avoid friendship with the design you have, the better option is to design it out. In the end, the design you have either requires inheritence combined with the friendship system you have, or devout following of the rule of 3/rule of 5 implementing reference counting semantics. None of these are great options, such inheritance patterns invade code that ought to be able to ignore that it has a ref count component, and worse, you can't use your resource manager with pre-existing classes. The "proper" solution is to invert the relationship, and then you end up with shared_ptr, essentially -- shared_ptr and friends are basically just boxes you can put things in to take care of lifetime management.

In general, you should always be suspicious of 'friend' relationships -- friend relationships in C++ are actually more tightly coupled than inheritance relationships, it breaks encapsulation entirely (no contract can be enforced upon the friend, its 100% trust, unlike inheritance) and forces the 'base' class (the one from which friendship is declared) to know about (and be modified to suit) any class that would need access to its internals. This is why I called this arrangement "intrusive, heavy, and brittle".

Even regular old inheritance, when not used strictly to express an is-a relationship, is suspicious -- less so than friendship -- but suspicious enough to take a careful look at.

October 01, 2014 10:48 PM
sirGustav

Alternatively you can have your resource manager store std::weak_ptrs, and call std::weak_ptr::lock() or check std::weak_ptr::expired(). If weak_ptr failed to lock or was expired, then it means it was de-allocated because of zero references (the resource manager's weak_ptr not counting as a reference), so you can then re-load it.

/..../

This could actually even be wrapped up as a single generic template function rather than a templated class or class hierarchy.

I've done exactly this. Works pretty well.

https://github.com/madeso/euphoria/blob/master/euphoria/cache.h

I noticed however that the create function probably doesn't need to be a parameter, should change that...

October 02, 2014 04:47 AM
apatriarca
When working on a game we have complete control on the resources we use. We can choose the file format s, the methods used to identify the various resources and how to represent relationships between them. We are not limited by external constraints like in other softwares. Moreover, resources are usually loaded and unloaded at very specific moment of time. We can indeed identify global, level or frame specific resources. There is really no need of fancy and abstract resource managers.

By only stating your objectives and no motivation behind them it seem you are trying to solve an abstract and non-existent problem. In my opinion, the first thing an article like this should do is presenting the problem and motivating the decisions made in the implementation and design of the solution.
For example:
1. Why do you think you need reference counting? Have you considered other alternatives? How bad is resource duplication and late deallocation is in your system? Have you considered the possibility a resource is unloaded just before it is requested again?
2. Why you think you can't control when a resource is deallocated and you need an automatic system to do this? For example, if you have resources that should be alive for an entire level, you may decide you can simply deallocate all of them at the end of the frame.
3. Why do you think you need strings to access your resources? String maps are not fast and resource names often differ only in small parts. Have you tried other alternatives (like using IDs)?.
4. Why do you need a map to maintain the relationships between resources? Why resources can't do it by themselves?
October 02, 2014 11:00 AM
Servant of the Lord

I noticed however that the create function probably doesn't need to be a parameter, should change that...

I agree it should either be a parameter, or a templated or overloadable function. But perhaps the default version of the templated function should pass the key and settings directly into the constructor of the object.

October 03, 2014 12:06 AM
Phredo

Singleton is one of the biggest problems with 'manager' classes, so the author has done well to avoid it. I'll raise your chime-in to say that relying on a resource base class is intrusive, heavy, and brittle, and that the code could be simplified by using C++11 smart pointers to handle lifetime management and ownership.

I will chime-in and say I am happy at least to see it was not implemented as a Singleton.

I'm curious.... why the anti-singleton sentiment here? It would make plenty of sense to use it for something like this in that one single, global instance is pretty much what you'd want from a resource manager!

October 07, 2014 05:02 PM
Dave Hunt

It would make plenty of sense to use it for something like this in that one single, global instance is pretty much what you'd want from a resource manager!

Your assumption is one of the biggest reasons (outside of the technical/philosophical reasons) why a singleton is a bad idea. In the case of a resource manager, the user (developer) may want to have one for UI elements and another for per-level resources, for example.

Many cases of "there can be only one" are really more like "I personally don't see a need for more than one."

October 08, 2014 04:59 PM
Servant of the Lord

I'm curious.... why the anti-singleton sentiment here? It would make plenty of sense to use it for something like this in that one single, global instance is pretty much what you'd want from a resource manager!

A single global instance? Fine. Make it a global. Or a global unique_ptr if you need to delay its construction until after main() is entered.

But enforcing a single instance of it for no reason? Singletons are for putting a lock on an object preventing it from being constructed more than once.

Singletons don't say, "I only need one.", singletons say, "Make absolutely sure that there can only ever be one."

Singletons enforce that there is a 'single' instance. Some things are genuinely useful as globals: Like logging. But that doesn't mean you want to enforce that there is only one logging instance - you might want separate instances of loggers for separate subsystems (for example).

std::cout and std::cin are both globals. Neither are singletons, which is good. What if you want a separate std::cout for error output instead of just general output? Well, create another global called std::cerr, and... oh wait, they already did. If std::cout was a singleton, std::cerr couldn't exist (unless they copy+pasted the code of that singleton, instead of just creating a second instance of it).

The problem is not really truly with singletons (there are very few situations where they are useful, but those rare situations do exist), the problem is most people abuse singletons when they really just want a global (whether initialized before or after main() enters). A secondary semi-related issue is over-using (non-const) globals out of laziness or poor design (or genuine lack of time when being rushed on a project), but that wasn't being commented on.

October 13, 2014 08:05 PM
Phredo

It would make plenty of sense to use it for something like this in that one single, global instance is pretty much what you'd want from a resource manager!

Your assumption is one of the biggest reasons (outside of the technical/philosophical reasons) why a singleton is a bad idea. In the case of a resource manager, the user (developer) may want to have one for UI elements and another for per-level resources, for example.

Many cases of "there can be only one" are really more like "I personally don't see a need for more than one."

Well it's not really an assumption as I've had plenty of experience in implementing various systems and I just go with what has worked best (for me). Clearly it's a matter of personal tastes. I like the idea of a single, unified, global instance of a system that has a set responsibility BUT there are times when multiple instances of a system can use quite useful too. Mix and match!

So how would multiple instances of a resource manager handle the likely case of the same asset being used by different sub-systems? Does each resource manager load it's own assets even if that means data duplication? Are all resource managers made global so you can access each one as you need it and if so, then why not just make one instance that can better manage ALL assets as a whole set?

To me a resource manager (amongst other important systems) isn't the case of "there can be only one"... it's "there SHOULD BE only one!" as it's a system that is going to be accessed EVERYWHERE in the game where as a system like an entity manager is more than likely only going to be used in the gameplay section.... but even then why limit yourself right!? :)

October 14, 2014 06:16 PM
Phredo

I'm curious.... why the anti-singleton sentiment here? It would make plenty of sense to use it for something like this in that one single, global instance is pretty much what you'd want from a resource manager!

A single global instance? Fine. Make it a global. Or a global unique_ptr if you need to delay its construction until after main() is entered.

But enforcing a single instance of it for no reason? Singletons are for putting a lock on an object preventing it from being constructed more than once.

Singletons don't say, "I only need one.", singletons say, "Make absolutely sure that there can only ever be one."

Singletons enforce that there is a 'single' instance. Some things are genuinely useful as globals: Like logging. But that doesn't mean you want to enforce that there is only one logging instance - you might want separate instances of loggers for separate subsystems (for example).

std::cout and std::cin are both globals. Neither are singletons, which is good. What if you want a separate std::cout for error output instead of just general output? Well, create another global called std::cerr, and... oh wait, they already did. If std::cout was a singleton, std::cerr couldn't exist (unless they copy+pasted the code of that singleton, instead of just creating a second instance of it).

The problem is not really truly with singletons (there are very few situations where they are useful, but those rare situations do exist), the problem is most people abuse singletons when they really just want a global (whether initialized before or after main() enters). A secondary semi-related issue is over-using (non-const) globals out of laziness or poor design (or genuine lack of time when being rushed on a project), but that wasn't being commented on.

When used right, they are a very amazing and effective tool! Plenty of heavy commercial engines like Unreal or Ogre use them for a reason! I also agree they aren't always used right and very much abused!

I'd absolutely want only once instance of a render system or resource manager to be active at anyone time. I'd want to know that when I ask a resource manager for an asset that it's the same instance that I used when I loaded an asset into it earlier on.... hence the "enforcement" aspect of them.

In a single coder environment (as is the case for me now), I could see how using global pointers could be ok if you didn't want to go with singletons BUT when I worked for a large company with MANY coders, closely working away together on the same project, I can't tell you how many times singletons avoided so many problems where unprotected global instances caused them. Coders were (intentionally or not) duplicating instances of major systems and using the wrong instances of them. Coders are humans and humans aren't robots thus we will make mistakes! :)

October 14, 2014 06:36 PM
apatriarca
The only reason to "enforce" something is to prevent something bad to happen. And it is rarely the case for singletons. There may be several reasons to have more than one resource manager. You may for example want to separate assets by their lifespan or by type or by usage or by .. If you decide to use a singleton early in the development, you are limiting your ability to make modifications and extensions later on.

A singleton is also often more difficult to implement, maintain and debug than most alternatives. You do not need to use a singleton to prevent copies of a global variable or to delay its initialization for example. unique_ptr s or private copy constructors or opaque interfaces are simpler and more flexible solutions to these problems.

Finally, you are making everything a lot more verbose.
October 15, 2014 09:51 AM
Phredo

Right! You are defensively coding to prevent bad things from happening and there are many tools at your disposal for such tasks... one of which is the singleton!

With respect to the case of multiple resource managers.... I ask again what I asked above:

How would multiple instances of a resource manager handle the likely case of the same asset being used by different sub-systems? Does each resource manager load it's own assets even if that means data duplication? Are all resource managers made global so you can access each one as you need it and if so, then why not just make one instance that can better manage ALL assets as a whole set?

In this case, you'd want a resource manager that does all the heavy lifting behind the scenes such as managing life span of assets based on types or usage. You'd also want that manager accessible from anywhere in the game so there is the case for a global variable. A singleton provides that along with extra defensive measures.

Singletons are quite easy to implement. I use a templated version which works quite well and once implemented, you don't ever need to debug them again. Singletons basically came about as a way to consolidate all the various tricks you just mentioned in order to enforce a single instance of an object but in a cleaner way.

October 16, 2014 07:38 PM
apatriarca

With respect to the case of multiple resource managers.... I ask again what I asked above:
How would multiple instances of a resource manager handle the likely case of the same asset being used by different sub-systems? Does each resource manager load it's own assets even if that means data duplication? Are all resource managers made global so you can access each one as you need it and if so, then why not just make one instance that can better manage ALL assets as a whole set?

Each resource manager should clearly have its own purpose.. You do not create a bunch of instances simply because you can do it. There should be a reason to create them. The reply to your questions depends on this reason.

You may have various resource managers and no duplicates. You simply have to define a criteria to choose what resource manager to use depending on the asset. For example you may have a resource manager for sounds and one for images. You do not have duplicates and you always know what resource manager to use for what asset. But you may also decide duplicates are not a problem and choose a different design.

You may also have resource managers with different implementations. In this case you may optimize the resource manager depending on the usage of the assets. For example, if you have a resource manager for level-specific assets you may decide all assets are only loaded at the beginning of the level and released at the end. You may thus decide to create a resource manager with load everything at initialization and release everything at destruction. Or maybe you have a huge level and decide some resources should be loaded as needed. In this case you may use a resource manager for the assets which should be loaded in this way and more static assets which should be in memory for the entire level.

In this case, you'd want a resource manager that does all the heavy lifting behind the scenes such as managing life span of assets based on types or usage. You'd also want that manager accessible from anywhere in the game so there is the case for a global variable. A singleton provides that along with extra defensive measures.

Maybe this is not what you want in your design. You may for example decide globals are problematic in your multi-threaded engine and want to avoid them. You may have well separated sub-system and one resource manager for sub-system and thus not need to make the manager accessible everywhere. And maybe you do not want the restrictions of singletons.

Singletons are quite easy to implement. I use a templated version which works quite well and once implemented, you don't ever need to debug them again. Singletons basically came about as a way to consolidate all the various tricks you just mentioned in order to enforce a single instance of an object but in a cleaner way.

IMHO Implementing a singleton which correctly work with multiple threads is not easy. But you are right that you can copy the implementation each time you create one. I don't think a singleton is cleaner than the other solutions. In fact, I prefer to write something like do_something(...) than Class::getInstance()->do_something(..). You can use a temporary variable to store the pointer, but it is still more code each time you use the class without any particular advantage.
October 16, 2014 11:36 PM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!

Featured Tutorial

A resource manager for handling game assets in your games.

Advertisement

Other Tutorials by TheItalianJob71

TheItalianJob71 has not posted any other tutorials. Encourage them to write more!
Advertisement