This was introduced with the /tmp on tmpfs by default misfeature unfortunately.

Firefox cannot have the TMPDIR on tmpfs because the user usually expects it to use the TMPDIR for very large files that can be downloaded.

Note that the Fedora breakage is not specific to Firefox. Also note that quite some users expect Firefox to download into $HOME/Downloads anyway, which adds even another default location (xdg-user-dir DOWNLOAD) which only confuses matters further. Users, who don't like the default /var/tmp, need to override $TMPDIR, and not even that works completely with Fedora.

to add some basic information here:

gnome-terminal is launched via d-bus activation nowaday (see /usr/share/dbus-1/services/org.gnome.Terminal.service). Thus it is getting the same environment as all the other dbus activated applications; which is an increasing number.

BZ ticket 956380 examined g_get_tmp_dir() and its defaults -> multiple different variables which further
complicate the matter. Nowhere does it use /var/tmp as a default. Firefox should not introduce its own
default and especially not export it to programs it starts.

This is not true anymore. Since

https://git.gnome.org/browse/glib/commit/?id=c2352409051df49a08d9dced5b69ede434e90723

g_get_tmp_dir only looks at TMPDIR

Replying to [comment:1 tmraz]:

This was introduced with the /tmp on tmpfs by default misfeature unfortunately.

I don't think there is any relation between these two things. What makes you think there is ?

Previously the /tmp was sufficient as default for all the various applications. That changed with that above-mentioned misfeature. I do not plan to discuss this thing further here though.

Pragmatically, by far the easiest solution seems to be to patch claws-mail to hardcode a directory it uses for the single-instance state.

Ideally, I think we should:
* Define a new public API for getting temporary directories (or perhaps skip that and go straight to temporary, optionally anonymous files!); where “temporary” is defined by semantics/properties and not by location/method of determining the value; so that applications know what to use the directory for but never try to second-guess where that directory is located.
* (Which directly leads to a question “API where”? GLib is a reasonable place but some projects categorically refuse to depend on it; but this all will only work long-term if it is an API and not a convention with multiple implementations.)
* (Note that “a place to put single-instance state” is one of the more rare uses for temporary files. Arguably this should be considered “runtime state” and put e.g. somewhere in ~/.local .)

• Implement that API, driven by an appropriate configuration option. Given all the bugs cited here, environment variables are problematic (unless the implementation went to the extreme lengths of hiding the environment variable from environ/execve completely), so I suppose this would mean either a hard-coded path or a config file.

This would probably be a first case of FESCo driving API design like this (or even asking for someone to drive an API design). I would ''love'' to move FESCo to that direction, but I suspect not everyone feels that way, and there are significant practical/coordination difficulties with FESCo doing this kind of thing at the moment.

g_get_tmp_dir only looks at TMPDIR

The bugzilla ticket I referred to is a few months older than the commit you've pointed at. ;-)

Pragmatically, by far the easiest solution seems to be to patch claws-mail to
hardcode a directory it uses for the single-instance state.

Will you try to convince upstream? Because if it doesn't happen upstream, it will be a problem:
https://bugzilla.redhat.com/956380#c9

Replying to [comment:9 mschwendt]:

Pragmatically, by far the easiest solution seems to be to patch claws-mail to
hardcode a directory it uses for the single-instance state.

Will you try to convince upstream? Because if it doesn't happen upstream, it will be a problem:
https://bugzilla.redhat.com/956380#c9

I’d say that ease of switching between Fedora packages and git HEAD builds is a low-priority concern. How many people are affected, a dozen at most? And overwhelmingly experienced enough to be able to apply the Fedora-specific patch on top of the checkout.

Getting this solved for the non-programming users who may not be able to follow all the curious aspects of TMPDIR propagation that you have identified, and thus see the software fail very randomly, seems far more important to me.

Setting $TMPDIR is really something we shouldn't do in codepaths we ship. It's a tool for admins and users, not for us, the OS vendor.

Mozilla should put "large" files in /var/tmp (i.e. everything basically unbounded, that might potentially grow to huge sizes, such as temporary downloads). "Small" files should be placed in /tmp. Claws should put any runtime socket/PID/whatever in $XDG_RUNTIME_DIR.

Replying to [comment:8 mitr]:

Pragmatically, by far the easiest solution seems to be to patch claws-mail to hardcode a directory it uses for the single-instance state.

It should just use XDG_RUNTIME_DIR, that's what it is for. It's exposed by glib, there's an fdo spec about it, and so on.

Ideally, I think we should:
* Define a new public API for getting temporary directories (or perhaps skip that and go straight to temporary, optionally anonymous files!); where “temporary” is defined by semantics/properties and not by location/method of determining the value; so that applications know what to use the directory for but never try to second-guess where that directory is located.

Note that we recently added something like this to libsystemd:

http://cgit.freedesktop.org/systemd/systemd/tree/src/systemd/sd-path.h#n27

And glib has parts of this already, too.

• (Note that “a place to put single-instance state” is one of the more rare uses for temporary files. Arguably this should be considered “runtime state” and put e.g. somewhere in ~/.local .)

No! runtime state should not be in the home directory, as AF_UNIX are generally not OK there. Use XDG_RUNTIME_DIR for this, which explicitly allows AF_UNIX sockets and stuff.

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables

Here is one point of view from a Claws Mail developer:
http://claws-mail.org/pipermail/users/2013-February/005423.html

Hardcoding a tmp dir in Claws Mail doesn't fix any other programs, that may be affected by the issue. Also, the the behaviour would need to match Firefox's wrapper script:
{{{
## Use $TMPDIR if set. Otherwise use /var/tmp instead of /tmp
## because of 1GB /tmp limit in Fedora 18 and later.
## See: https://bugzilla.redhat.com/show_bug.cgi?id=867073
##
TMPDIR="${TMPDIR:-/var/tmp}"
export TMPDIR
}}}

And as pointed out, evaluating $TMPDIR before falling back to a hardcoded default would not work fully either.

How many people are affected, a dozen at most?

Gimme a break! Do you want to increase the number of Fedora users or drive away further ones?

Replying to [comment:12 lennart]:

Ideally, I think we should:
* Define a new public API for getting temporary directories (or perhaps skip that and go straight to temporary, optionally anonymous files!); where “temporary” is defined by semantics/properties and not by location/method of determining the value; so that applications know what to use the directory for but never try to second-guess where that directory is located.

Note that we recently added something like this to libsystemd:

http://cgit.freedesktop.org/systemd/systemd/tree/src/systemd/sd-path.h#n27

And glib has parts of this already, too.

I’m worried two implementations only coordinated by convention is still a long-term failure.

Replying to [comment:13 mschwendt]:

Hardcoding a tmp dir in Claws Mail doesn't fix any other programs, that may be affected by the issue.

True. However, that TMPDIR override has been in Firefox for two years, and this is a first (or first high-profile?) breakage. Perhaps there are not that many other programs that use $TMPDIR like this.

Also, the the behaviour would need to match Firefox's wrapper script:

And as pointed out, evaluating $TMPDIR before falling back to a hardcoded default would not work fully either.
No, I proposed ''hard''-coding, explicitly to take all environment variables out of the equation.

How many people are affected, a dozen at most?

Gimme a break! Do you want to increase the number of Fedora users or drive away further ones?

I want to increase the number of users, that’s why I think favoring users that are not claws-mail developers (presumably many) over claws-mail developers and packagers (fewer) makes sense. That’s not to say that the claws-mail developers and packagers don’t matter, but that their interest should not stop us from fixing this for the others.

Completely hardcoding it (with no option for customisation) might turn up issues we're not aware of yet. Firefox at least can be customised via setting $TMPDIR. ;)

I've suggested $HOME/.claws-mail/... before - I wish the source code would explain why the file is stored in tmp instead.

Lennart's proposal about using XDG base dirs makes sense. I've forwarded it here:
http://lists.claws-mail.org/pipermail/users/2014-November/011324.html

Adding this to the agenda for tomorrow's IRC meeting. mschwendt and lennart and others, can you come to the meeting on Wednesday at 18:00UTC in #fedora-meeting please?

Replying to [comment:16 mschwendt]:

Completely hardcoding it (with no option for customisation) might turn up issues we're not aware of yet.

I can’t think of an example. For a comparison, we never had a mechanism for customizing the location of /var/tmp.

Lennart's proposal about using XDG base dirs makes sense. I've forwarded it here:
http://lists.claws-mail.org/pipermail/users/2014-November/011324.html

Note that using XDG base dirs has exactly the same structural problems, relying on environment variable passing to work. It does have the advantage of not using a variable that we know to be frequently overriden (by Firefox and users) in practice.

Replying to [comment:18 mitr]:

I can’t think of an example. For a comparison, we never had a mechanism for customizing the location of /var/tmp.
Sorry, I meant /var/lock .

Note that using XDG base dirs has exactly the same structural problems,
relying on environment variable passing to work.

Yet that currently works by default. Also tested with Openbox.

I can’t think of an example.

I can imagine plugins/extensions not provided by Fedora, which would look for runtime state files in the unpatched location. Consistency (also with upstream) would be a plus.

Btw:

Fedora 21: Firefox by default downloads Fedora-Live-Workstation-x86_64-21_Beta-4.iso (plus a temporary *.iso.part file!) to $HOME/Downloads instead of /var/tmp.

Fedora 18: Also!

So, not even that Fedora patch does what has been explained in the bugzilla ticket. It does not affect downloads. At least not when running GNOME. According to the source: malloc page files (not enabled by default in the source), database files, ... some parts of the source code give $TMP and other env vars higher priority than $TMPDIR.

Firefox experts may chip in and tell what "large files" would be affected. The theory behind this patch is breaking apart.

{{{

$ cd mozilla-release

$ grep TMPDIR * -R|grep .c\:

db/sqlite3/src/sqlite3.c: if( !azDirs[1] ) azDirs[1] = getenv("SQLITE_TMPDIR");

db/sqlite3/src/sqlite3.c: if( !azDirs[2] ) azDirs[2] = getenv("TMPDIR");

db/sqlite3/src/sqlite3.c: 0, / getenv("SQLITE_TMPDIR") /

db/sqlite3/src/sqlite3.c: 0, / getenv("TMPDIR") /

db/sqlite3/src/sqlite3.c: if( !azDirs[0] ) azDirs[0] = getenv("SQLITE_TMPDIR");

db/sqlite3/src/sqlite3.c: if( !azDirs[1] ) azDirs[1] = getenv("TMPDIR");

grep: media/webrtc/trunk/webrtc/tools/e2e_quality/audio/perfipc/chromium/src/base/file_util_posix.cc: const char* tmp = getenv("TMPDIR");

js/src/ctypes/libffi/src/closures.c: { open_temp_exec_file_env, "TMPDIR", 0 },

No such file or directory
memory/mozjemalloc/jemalloc.c: * 1) MALLOC_TMPDIR
memory/mozjemalloc/jemalloc.c: * 2) TMPDIR
memory/mozjemalloc/jemalloc.c: if ((s = getenv("MALLOC_TMPDIR")) == NULL && (s =
memory/mozjemalloc/jemalloc.c: getenv("TMPDIR")) == NULL)
nsprpub/pr/tests/testfile.c:#define TMPDIR_LEN 64
nsprpub/pr/tests/testfile.c:char testdir[TMPDIR_LEN];
nsprpub/pr/tests/testfile.c: WCHAR tdir[TMPDIR_LEN];
nsprpub/pr/tests/testfile.c: len = GetTempPath(TMPDIR_LEN, tdir);
nsprpub/pr/tests/testfile.c: if ((len > 0) && (len < (TMPDIR_LEN - 6))) {
nsprpub/pr/tests/testfile.c: WideCharToMultiByte(CP_ACP, 0, tdir, -1, testdir, TMPDIR_LEN, 0, 0);
nsprpub/pr/tests/testfile.c: len = GetTempPath(TMPDIR_LEN, testdir);
nsprpub/pr/tests/testfile.c: if ((len > 0) && (len < (TMPDIR_LEN - 6))) {
security/nss/lib/softoken/sdb.c: azDirs[1] = getenv("TMPDIR");
security/nss/lib/dbm/src/h_page.c: tmpdir = getenv("TMPDIR");
security/nss/lib/sqlite/sqlite3.c: if( !azDirs[1] ) azDirs[1] = getenv("TMPDIR");
security/nss/cmd/selfserv/selfserv.c: tmp = getenv("TMPDIR");
security/nss/cmd/httpserv/httpserv.c: tmp = getenv("TMPDIR");
toolkit/crashreporter/google-breakpad/src/third_party/glog/src/logging.cc: env = getenv("TEST_TMPDIR");
toolkit/crashreporter/google-breakpad/src/third_party/glog/src/logging.cc: getenv("TEST_TMPDIR"),
toolkit/crashreporter/google-breakpad/src/third_party/glog/src/logging.cc: getenv("TMPDIR"), getenv("TMP"),
}}}

Replying to [comment:20 mschwendt]:

Fedora 21: Firefox by default downloads Fedora-Live-Workstation-x86_64-21_Beta-4.iso (plus a temporary *.iso.part file!) to $HOME/Downloads instead of /var/tmp.

Fedora 18: Also!

So, not even that Fedora patch does what has been explained in the bugzilla ticket. It does not affect downloads. At least not when running GNOME. According to the source: malloc page files (not enabled by default in the source), database files, ... some parts of the source code give $TMP and other env vars higher priority than $TMPDIR.

Firefox experts may chip in and tell what "large files" would be affected. The theory behind this patch is breaking apart.

Martin, could you help us with this, please?

If you have Firefox configured with "Always ask me where to save files", and you click on a link, Firefox will immediately start downloading it to /var/tmp while showing you the file save dialog.

change Firefox's default temp dir from /tmp to /var/tmp
firefox-default-temporary-dir-to-var-tmp.patch

IMHO the above (untested) patch to Firefox would be a better than the exporting of TMPDIR in the shell wrapper.

If you have Firefox configured with "Always ask me where to save files",
and you click on a link, Firefox will immediately start downloading it
to /var/tmp while showing you the file save dialog.

Cannot reproduce.

I've moved away ~/.mozilla, started firefox, opened preferences, switched the radio button to "Always ask me where to save files", then started a download. The file manager dialog opened in ~/Downloads.

With a fresh user account in Fedora 21, also the Firefox Preferences dialog displays ~/Download as default location.

What the dialog shows is irrelevant. The point is that before the user makes any selection in that dialog, Firefox is already downloading the file in the background and saving it as /var/tmp/*.part.

Here it doesn't. Seems there is blackbox behaviour in various areas. ;)

Here it opens the file manager, and no download is taking place in the background. I've waited long enough and checked all possible directories. Download just didn't start. Fedora 21 testing.

Meanwhile I've also checked the special scenario of a customised $XDG_RUNTIME_DIR env variable. gnome-terminal's mailto handler also deletes it. Again, one could argue how many users would be affected, and so on ... it just wouldn't be a safe solution either.

Okay, it took more time before download started with a strange file name (eF7CnzjP.iso.part) in /var/tmp.

I have now built Firefox with firefox-default-temporary-dir-to-var-tmp.patch, dropped the TMPDIR hack from the shell wrapper and tested it. The downloads go to /var/tmp as expected.

I read the log from today's FESCo meeting. At one moment mschwendt said:

Firefox has been fixed, tells the ticket.

This is a misunderstanding of my previous comment. I merely proposed a patch for Firefox. I have tested it using a local build. I am not the maintainer of Firefox. Sorry for the ambiguity.

Okay, thanks for pointing that out.

https://bugzilla.redhat.com/1161110 - /usr/bin/firefox should not mess with TMPDIR

As an addition to the comments on hardcoding the socket file location in Claws Mail, that would also need to happen in "clawsker", which is a separate program written in Perl to access extra/hidden configuration settings. => Another program where this ought to be done upstream, if at all.

We discussed this again at todays meeting:

AGREED: FESCo does not currently want to mandate $TMPDIR being reliably consistent throughout an user session. If claws-mail requires a consistent place to use, it should use a different mecahnism. (+6,0,0)