Where to put data under Vista?

Author Published on 18 Comments

Running into the problem

I recently needed to figure out where to put datafiles when upgrading a client project. Part of the upgrade is to make the application Vista compatible. The files were stored in the application folder which was by default installed into Program Files. Under Vista that will not work because Program Files is read-only to all but administrator accounts.

When we released our Build Automator in May 2008 I had spent some time doing research on how best to deal with data file placement under Vista. Of course I had managed to forget most of that so I decided to dig into this again and write down what I found so I could have a future reference of what the conclusions were.

Vista's UAC

With Vista's User Access Control (UAC) Microsoft changed the way user access is granted. By default user accounts on XP could do pretty much whatever they wanted - they had administrator privileges by default. Under Vista this is not so. Regular user account does not have administrator rights. Even an administrator account does not have administrator privileges to everything and will be prompted to elevate to an administrator privileges for certain actions.

What is CSIDL?

To get a good view of the standard folders on a computer you can download our Special Folder program for free. When you run it, it will show you all the different special folders on your computer. The Special folders are generally referred to with various CSIDL names, such as CSIDL_Desktop, which is the desktop folder for the user or CSIDL_Personal which is the "My Documents" folder under XP and "Documents" folder under Vista. CSIDL stands for "constant special item ID list". For more information about individual CSIDL, visit this link.

In Vista, Microsoft changed the names of CSIDL to KnownFolderID. For more information about the KnownFolderIDs see:

http://msdn.microsoft.com/en-us/library/bb776911(VS.85).aspx
http://msdn.microsoft.com/en-us/library/bb762584(VS.85).aspx

The CSIDL and the shell API calls that work on XP and previous versions of Windows work just fine under Vista. On the the other hand, the KnownFolderIDs and it's shell API functions will not work under XP as the API calls require shell version 6.0.6000 or later. There are various API calls that can be used to get the folder information. Our Icetips Utilities contain simple methods to extract the folder information and at the end of this document I have put some simple code to use for this.

Where to put those pesky data files?

Location of data files on Vista has been a constant headache for developers. Should they be placed in CSIDL_COMMON_APPDATA, CSIDL_LOCAL_APPDATA, CSIDL_COMMON_DOCUMENTS or CSIDL_PERSONAL

Where exactly to place the files really depends on the nature of the data in my opinion. Is it for just the currently logged in user or is it for anyone on the computer? That may also need to be resolved when software is installed, i.e. is it for all users or just the user installing the software. That, in general, is the difference between the LOCAL and COMMON places. Just note that Microsoft did not use that naming convention on all the CSIDL values since for example CSIDL_PERSONAL is the LOCAL version of CSIDL_COMMON_DOCUMENTS!

Accessibility is another difference between those 4 folders. The CSIDL_COMMON_APPDATA works fine, but this folder is marked as hidden, and data in it is read-only to users that are not administrators or power users. That means that if users ever need to look for something in this folder, for example to e-mail a file to the developer, they may not be able to see it! It also can't be used by data that the application needs to write to, but is fine for data that it just needs to read. CSIDL_COMMON_DOCUMENTS is always visible. If the user never has to do anything with the files, including writing to them, CSIDL_COMMON_APPDATA is a perfect place to put it, but it gives all users on the PC read-only access to it.

Roaming

Vista, and older Windows versions also, have an option called Roaming. Several of the CSIDLs have Roaming counterparts. The roaming folders include for example the Start menu folders, cookies and network and printer shortcuts and CSIDL_APPDATA. Generally you should avoid the roaming folders unless you need to use them and you know what they mean and how to use them. Microsoft has an excellent document online about managing roaming and data deployment to roaming folders.

What to put in each folder?

Now let's explore the pros and cons of the four alternatives that we have, CSIDL_COMMON_APPDATA, CSIDL_LOCAL_APPDATA, CSIDL_COMMON_DOCUMENTS or CSIDL_PERSONAL.


CSIDL_COMMON_APPDATA


Vista: C:\ProgramData
XP: C:\Documents and Settings\All Users\Application Data

Suitable for application specific data that can be shared with all users and is provided by the developer and should only be updated by the administrator account or an elevated applictation. This folder is hidden so it is not visible in Explorer unless its settings are changed to show hidden and system files. All users have access to the data.

Of coures if you don't care if the user can view the datafiles, you can just as easily install them into CSIDL_LOCAL_APPDATA, but you still have the problem with the installer not knowing what folder is the actual user CSIDL_LOCAL_APPDATA. The perfect solution is to install default datafiles into CSIDL_COMMON_APPDATA and then let the application create a copy of those files in a matching folder structure in the CSIDL_LOCAL_APPDATA where each user has their own copy to start with.


CSIDL_LOCAL_APPDATA


Vista: C:\Users\UserName\AppData\Local
XP: C:\Documents and Settings\User Name\Local Settings\Application Data

Suitable for application data that is completely contained within the application as can be read and written to. This folder is not hidden and the user can explore it at will. Only the currently logged in user has access to this folder.

This folder poses a problem for the installer because when it runs under the administrator account it does not know what the currently logged in user is. The trick above for CSIDL_COMMON_APPDATA applies, where you install the data into the common application data, then copy it from there to the local folder. Possible problem here is that the uninstaller may not be able to uninstall the copied files so they may be left behind - may not be a problem at all, but worth noting.


CSIDL_COMMON_DOCUMENTS


Vista: C:\Users\Public\Documents
XP: C:\Documents and Settings\All Users\Documents

Suitable for data that may need to be access by the users with both read and write access. All users have write access to this folder so it can be used for data sharing or file sharing between users.


CSIDL_PERSONAL


Vista: C:\Users\User Name\Documents
XP: C:\Documents and Settings\User Name\My Documents

Suitable for data that may be access by the user. Only the currently logged in user has access to this folder. Since this is the "Documents" or "My Documents" folder any folders that are placed here are immediately visible to the user when (s)he opens the documents folder. Lot of software uses this folder as a suggested place to put new projects (Visual Studio, Setup Builder and our Build Automator for example). There are certain risks involved in placing critical data here. For example if the user is cleaning up it is not difficult to imagine some folders being accidentally deleted. On the other hand, this folder is probably the most likely to be backed up since it normally contains valuable documents!

Put it where it belongs

In general, CSIDL_COMMON_APPDATA is the safest location for general database files that the user should never mess with, including write to. CSIDL_LOCAL_APPDATA should generally be used for data that the user needs to use. By using the common folder to install into and letting the application copy files to the local folder as needed you have a system that works very well. For data that may need more user access consider using the CSIDL_COMMON_DOCUMENTS for shared data and CSIDL_PERSONAL for user specific data.

Note that you can, from your program, open an explorer window for this location even though it is hidden - just like SpecialFolders does. That way your program can give the user access to explore this folder if they need to. You could perhaps supply a small exe that does only that - explore the data folder. If you are using our Icetips Utilities, all you would need in such a program is:

ITS ITShellClass
Code
ITS.ITShellExec(ITS.GetSpecialFolder(IT_CSIDL_COMMON_APPDATA),IT_SE_Explore)

This will open an Explorer window on the ProgramData folder under vista. If you wanted to open it to a subfolder you would simply use something like this:

ITS ITShellClass
Code
ITS.ITShellExec(ITS.GetSpecialFolder(IT_CSIDL_COMMON_APPDATA) & '\Icetips Creative',IT_SE_Explore)

This way you can easily provide a very simple solution to your users to explore the otherwise hidden ProgramData folder. However, you should only do this if you have to, this simply shows you a way to deal with it if the need comes up. Of course you could use different methods, such as a program that emails you the files for investigation or does something else. I'm simply pointing out that just because this folder is hidden from the user by default in Vista it should not discourage you to use it to store application data!

Now that we have looked through the pros and cons of each of those folders, I hope this entry will help you to decide which one to use. I also hope it explains a little bit about why you should use this folder rather than the other.

Our Icetips Utilities mentioned here is a collection of classes and templates for Clarion developers. They are part of our Gold Subscription plan.

The Build Automator is a developer tool to automate compiling and building applications and installs. It works with many developer tools such as Visual Studio, Delphi/RAD Studio 2008, Clarion, SetupBuilder, Inno Setup, Setup Factor and MSI Factory.

--Arnor Baldvinsson

18 thoughts on “Where to put data under Vista?”

  1. Arnor
    Re CSIDL_COMMON_APPDATA and you say
    "All users have access to the data." then maybe you could clarify that this is only ReadOnly access for non-Admin non-Power users unless you use the installer to give them more rights.

    On an MS www site they say...
    "If an application requires normal Users to have write access to an application specific subdirectory of CSIDL_COMMON_APPDATA, then the application must explicitly modify the security on that sub-directory during application setup."

    John

    ↓
  2. Arnor Post author

    John,

    Thanks for the comment:)

    You are absolutely correct! I have modified the text to keep that firmly in mind and also how to work around that during installation. I've also updated the text on CSIDL_COMMON_DOCUMENTS which is a good place to use for shared read/write data.

    My way of dealing with this is to install data into CSIDL_COMMON_APPDATA and then use our SetupBuilderClass from the Icetips Utilties to copy it to CSIDL_LOCAL_APPDATA.

    You need a lot of coffee to go through this and get it all straight and apparently I was suffering coffee deficiency:)

    ↓
  3. Ben

    Hi

    What about data that has to be avaialble over a network? Most of our apps are used by multiple users in a small office environment (less than 10 users), connecting to the same data set.

    Because of the amount of users, it is not feasable to set up a dedicated server, but one of the user machines is used for that ....

    Kind Regards

    Ben

    ↓
  4. Arnor Post author

    Ben,

    That is a very good question. As far as I can tell CSIDL_COMMON_DOCUMENTS can be shared and used accross a network providing that the user can log into the Vista machine. So, apart from the sharing, the same basically applies as with single or multi user PC. That said you could also create a separate folder in the root and share that and either map it on the other network computers or use the UNC folder/filenames to connect to it.

    Best regards,

    Arnor

    ↓
  5. Shaun Hollingworth

    The whole thing is complete LUNACY.
    Programs ought to be able to have write access to their OWN folders, when they are running. If an installer needs to put initial data for a none elevated program, which that program needs to delete, later where does it put it ?

    The only place on Vista seems to be:
    One: The Documents folder where all uses will see it, when they browse "Documents"
    Two: On a temporary folder off a root disk. Not generally visible to users but not recommended either.

    Scenario:

    An installer wants to pass an application, a licence file, which the application then registers and deletes. It puts it in CSIDL_COMMON_APPDATA but then the software cant delete it.

    So it instead puts the file in CSIDL_COMMON_DOCUMENTS where a user spots the file before the software got a chance to run, and says "What's this" and promptly deletes the file. Ho Hum.

    What a complete mess.

    There should be a place data can be put, to be shared read-write, invisible to the user, without all this nonesense.

    ↓
  6. Arnor Post author

    Shaun,

    Easy: Place it into a subfolder for COMMON_DOCUMENTS or PERSONAL and then HIDE the subfolder. Now of course the user can see it if they turn on the option to see hidden folders, but I don't see this as any different from using Program Files. Anyone can go in there and delete data. Note that most software that I have already install a lot of their data into PERSONAL or COMMON_DOCUMENTS and it doesn't seem to cause them any problems:)

    ↓
  7. Paul

    Hi Arnor,

    Let's say you want to share documents among all of the users on a single computer. Based on the documentation, it looks like the recommended location would be CSIDL_COMMON_DOCUMENTS. However, a quick look shows that this folder isn't just shared among users of a single computer, it's also shared across the entire network.

    I'm looking for the place to share documents among users of a single computer, but NOT share the documents to the entire network. Seems like a simple request, but it doesn't seem to be one of the CSIDL options.

    Is the solution to have the install program open up a subfolder's security? I'd like to follow Microsoft's security protocol's if possible, but don't know what to do in this case.

    Thanks for the great article.

    Paul

    ↓
  8. Arnor Post author

    Hi Paul,

    CSIDL_COMMON_DOCUMENTS is not shared over the network. It, or folders in it, CAN be shared. For example I do not see or have access to any of the COMMON_DOCUMENTS on any of my machines on the network here (mix of Vista and XP) because none of them share any of the folders there.

    Hope this helps:)

    Arnor

    ↓
  9. Paul

    Thanks Arnor. You're right that one can turn on/off sharing for any folder. However, I'm guessing that you have not run the Network Connection Wizard to share ANY of your folders in XP. If you do, it will set the COMMON_DOCUMENTS folder as shared on the network. That is Microsoft's designated folder for network sharing.

    My software is widely distributed to many users. Some of my users will have network sharing turned on, and some will not. I'd like a common folder that can be used by all users of a single computer, yet will not be available to the network when network sharing is turned on.

    Again, there does not seem to be a CSIDL folder for this. I tend to agree with Shaun in his prior post. It's lunacy.

    Paul

    ↓
  10. Arnor Post author

    Paul,

    Correct, but it only does if the user shares it. I.e. if you go to the "Network and Sharing Center" in vista and turn public folder sharing ON. It could be argued that if you turn sharing of your public folders ON, that you want others (or yourself) to have access to them from other computers. If the user doesn't want other computers to see their data they can turn it off. I'm pretty sure it is turned off by default, at least it is turned OFF on all our vista machines.

    If you want to bypass it, there is always an option to place it into a subfolder in COMMON_APPDATA and modify security access to the folder. Or simply create a folder in the root and use that.

    Best regards,

    Arnor

    ↓
  11. pratik

    M$ continues to do stupid things w/ every version of Windows. let's face it, Vista should be a case study for even the "Right To Lifers".

    all this CSIDL non-sense now requires a thin dev staff to play games. shouldnt software developers be spending time developing solutions and not messing around w/ folders and where to place files of all things?

    doesnt his add unnecessary bloat to every app just to fig out where it's data is?

    i have reverted to the sensible CPD way of doing things. create a folder on the c: drive, install my pgm and dll's using a tried and true directory structure of

    c:\myCompany
    c:\myCompany\appname
    c:\myCompany\database

    since i deploy w/ Firebird this makes my life very easy as well as my customers.

    cant wait to see what Win7 brings i mean breaks 🙂

    ↓
  12. Arnor Post author

    Hi Pratik,

    While that works for some, I would not install software that installed itself into the root of a drive. I have not seen software that does that for many years. It's a simple case of picking a folder to place the data in and use it. There is no difference in installing to C:\mycompany or to CSIDL_COMMON_DOCUMENTS & '\mycompany' - it's just a folder location:) Common folder IDS, CSIDL, have been around since Windows 98 at least.

    Getting the appropriate CSIDL folder is simple question of a single API call to get the foldername. In my apps and client projects I just use a single procedure called GetDataPath() that returns the correct path. All files have a variable name and this is set when the program starts up, one call to GetDataPath() which makes a single call to the GetSpecialFolder() method of my ITShellClass:

    GetDataPath(),String
    ITS ITShellClass
    Code
    Return(ITS.GetSpecialFolder(IT_CSIDL_COMMON_DOCUMENTS) & '\myCompany\database')

    That's it:) Each file variable then uses GetDataPath() like this:

    Glo:File:Filename = GetDataPath() & '\myfile.tps'

    You can accomplish the same with PROP:Name etc. etc.

    Best regards,

    ↓
  14. Arnor Post author

    Hi Dinart,

    Note that I'm talking about WINDOWS explorer, not INTERNET explorer:) I use Total Commander for all file management.

    Best regards,

    Arnor

    ↓

  15. Making App Data UAC (Vista and Windows 7) safe and XP compatible

    http://profileexchanges.com/blog/?p=120

    The CHM version has been greatly enhanced !

    The reference material is useful to developers in general, regardless of programming languages or tools of choice.

    Some of the information is based on Clarion Third Party Add-on products and vendors for SoftVelocity, Inc Programming Tools (Clarion Enterprise and Professional Edition).

    Some new and updated sections include:

    CSIDL Reference topic

    ↓
  17. Flavio

    Hi, Arnor

    Re: CSIDL_LOCAL_APPDATA, you said:
    "Suitable for application data that is completely contained within the application as can be read and written to. This folder is not hidden and the user can explore it at will. Only the currently logged in user has access to this folder.
    This folder poses a problem for the installer because when it runs under the administrator account it does not know what the currently logged in user is."

    Since 5.2.0 (2007-09-19) version, INNO Setup has the runascurrentuser and runasoriginaluser flags.
    From http://www.jrsoftware.org/files/is5-whatsnew.htm :
    - On Windows Vista, [Run] section entries with the postinstall flag no longer inherit Setup's elevated privileges by default, and instead now execute with the (normally non-elevated) credentials of the user that started Setup initially. (There are some exceptions; see the runasoriginaluser flag documentation for details.)
    - Added new [Run] section flags: runascurrentuser and runasoriginaluser. These control which user credentials are used on Windows Vista when spawning processes. (runasoriginaluser is the default when the postinstall flag is used; runascurrentuser is the default otherwise.)

    ↓
  18. Arnor Post author

    Hi Flavio,

    I also believe that Setup Builder 7 can do this. Normally an elevated program cannot write to the "current user" profile since it is not the same user as the administrator user running the installer. I recall that Friedrich said he was testing this about a year ago, or so, but I haven't followed up on it.

    Best regards,

    ↓

Leave a Reply