Python tempfile: Gotchas I Learned the Hard Way

I spent an embarrassing amount of time debugging temp file issues before I actually read the docs properly. Here's what I learned so you don't repeat my mistakes.

The NamedTemporaryFile delete Gotcha

This one got me. I wrote this code and couldn't figure out why it failed on my teammate's Windows machine:

import tempfile
import subprocess
 
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt') as f:
    f.write("config data")
    f.flush()
    # This works on Mac/Linux, FAILS on Windows
    subprocess.run(['cat', f.name])

The problem: on Windows, you can't open a file that's already open by another process. The NamedTemporaryFile keeps the file open, so the subprocess can't read it.

The Fix: delete=False

import tempfile
import os
 
# Create file but don't auto-delete
f = tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False)
try:
    f.write("config data")
    f.close()  # Close it first!
    
    # Now other processes can access it
    subprocess.run(['cat', f.name])
finally:
    os.unlink(f.name)  # Clean up manually

Python 3.12+ Solution: delete_on_close

If you're on 3.12+, there's a cleaner option:

import tempfile
 
with tempfile.NamedTemporaryFile(
    mode='w',
    delete=True,
    delete_on_close=False  # New in 3.12!
) as f:
    f.write("config data")
    f.close()
    # File still exists after close
    subprocess.run(['cat', f.name])
# Deleted when context manager exits

This is the best of both worlds: automatic cleanup but other processes can access the file.

TemporaryFile: When You Don't Need a Path

If no external process needs to access your temp file, use TemporaryFile instead:

import tempfile
 
with tempfile.TemporaryFile(mode='w+b') as f:
    f.write(b"temporary data")
    f.seek(0)
    data = f.read()
# No path, no filename, just gone

On most Unix systems, this creates a truly anonymous file—it's never even visible in the filesystem. Perfect for scratch space when processing data in memory.

# Text mode works too
with tempfile.TemporaryFile(mode='w+', encoding='utf-8') as f:
    f.write("text content")
    f.seek(0)
    print(f.read())

The key difference:

TemporaryFile → no .name attribute (well, technically it has one but it's not useful)
NamedTemporaryFile → has .name you can pass to other tools

TemporaryDirectory: The One I Use Most

For most tasks, I actually need a temp directory more than a temp file:

import tempfile
from pathlib import Path
 
with tempfile.TemporaryDirectory(prefix='build_') as tmpdir:
    work_dir = Path(tmpdir)
    
    # Create whatever files you need
    (work_dir / 'input.json').write_text('{"data": 123}')
    (work_dir / 'config.yaml').write_text('setting: true')
    
    # Run your process
    result = process_files(work_dir)
# Everything cleaned up, including nested directories

What I love about this:

Creates the directory with secure permissions
Deletes everything inside when done (no leftover junk)
Cleans up even if your code throws an exception

The ignore_cleanup_errors Parameter

This saved me once when a subprocess hadn't released a file handle:

import tempfile
 
# Won't raise even if cleanup partially fails
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
    # Your code here
    pass

Added in Python 3.10. Use it when cleanup failures shouldn't crash your app.

Low-Level Control: mkstemp and mkdtemp

Sometimes the context manager approach doesn't fit. Maybe you're creating temp files in a loop and managing them elsewhere. That's when you use the low-level functions:

import tempfile
import os
 
# mkstemp returns (file_descriptor, path)
fd, path = tempfile.mkstemp(suffix='.dat', prefix='data_')
try:
    # Option 1: Write using file descriptor
    os.write(fd, b"raw bytes here")
    os.close(fd)
    
    # Option 2: Convert fd to file object
    # (but be careful not to close twice!)
finally:
    os.unlink(path)  # YOU are responsible for cleanup

For directories:

import tempfile
import shutil
 
tmpdir = tempfile.mkdtemp(prefix='work_')
try:
    # Use the directory
    pass
finally:
    shutil.rmtree(tmpdir)  # YOU are responsible for cleanup

When I Use Low-Level Functions

File needs to outlive the current function
Managing a pool of temp files
Need the file descriptor for select() or other low-level I/O
Integration with C extensions expecting fd

Security: Why tempfile Matters

Before I understood temp file security, I did stuff like this:

import os
 
# DON'T DO THIS
path = f"/tmp/myapp_{os.getpid()}.tmp"
with open(path, 'w') as f:
    f.write(secret_data)

This has two security problems:

1. Race Conditions (TOCTOU)

# Time-of-check...
if not os.path.exists(path):
    # Attacker creates file/symlink here!
    with open(path, 'w') as f:  # Time-of-use
        f.write(secret_data)

An attacker could create a symlink at that path pointing to /etc/passwd or another sensitive file.

2. Predictable Names

If an attacker knows your file naming scheme, they can:

Pre-create the file to cause your app to fail or behave unexpectedly
Create symlinks to redirect writes to sensitive files
Read your temp data if permissions are wrong

The tempfile Solution

tempfile functions create files atomically with:

Random, unpredictable names
Secure permissions (0600 - owner only)
The O_EXCL flag (fails if file exists, preventing symlink attacks)

import tempfile
 
# SAFE: atomic creation, random name, secure permissions
with tempfile.NamedTemporaryFile(mode='w') as f:
    f.write(secret_data)

Where Do Temp Files Go?

import tempfile
 
print(tempfile.gettempdir())
# Linux: /tmp (or /var/tmp, depends on distro)
# macOS: /var/folders/.../T/
# Windows: C:\Users\...\AppData\Local\Temp

The TMPDIR Environment Variable

You can control temp file location:

# Shell
export TMPDIR=/my/custom/tmp
python my_script.py

Or in code:

import tempfile
 
# Check precedence: TMPDIR > TEMP > TMP > platform default
# Override for your process
tempfile.tempdir = '/custom/path'
 
# Now all temp files go there
with tempfile.NamedTemporaryFile() as f:
    print(f.name)  # /custom/path/tmpXXX

Per-File Override

import tempfile
 
# Just this file goes somewhere specific
with tempfile.NamedTemporaryFile(dir='/var/cache/myapp') as f:
    print(f.name)  # /var/cache/myapp/tmpXXX

This is useful when:

You need temp files on a specific filesystem (for atomic rename)
Some directories have more space or different cleanup policies
You're writing to a RAM disk for speed

Pattern: Processing Uploads

Here's how I handle uploaded files that need processing:

import tempfile
from pathlib import Path
import os
 
def process_upload(data: bytes, original_filename: str) -> dict:
    """Process uploaded file through external tool."""
    suffix = Path(original_filename).suffix
    
    # Create temp file (delete=False for Windows compat)
    with tempfile.NamedTemporaryFile(
        suffix=suffix,
        delete=False
    ) as f:
        f.write(data)
        temp_path = f.name
    
    try:
        # Now external tools can access it
        result = run_analysis_tool(temp_path)
        return result
    finally:
        # Always clean up
        os.unlink(temp_path)

For larger uploads where you want to stream:

import tempfile
import shutil
 
def handle_streaming_upload(upload_stream, chunk_size=8192):
    """Handle large uploads without loading into memory."""
    with tempfile.NamedTemporaryFile(delete=False) as f:
        temp_path = f.name
        for chunk in iter(lambda: upload_stream.read(chunk_size), b''):
            f.write(chunk)
    
    try:
        return process_large_file(temp_path)
    finally:
        os.unlink(temp_path)

Pattern: Test Fixtures

This is probably where I use tempfile the most:

import tempfile
from pathlib import Path
import pytest
 
@pytest.fixture
def work_dir():
    """Provide a clean temp directory for each test."""
    with tempfile.TemporaryDirectory() as tmpdir:
        yield Path(tmpdir)
 
def test_file_processor(work_dir):
    # Setup
    input_file = work_dir / 'input.txt'
    input_file.write_text('test content')
    
    output_file = work_dir / 'output.txt'
    
    # Exercise
    process_file(input_file, output_file)
    
    # Verify
    assert output_file.read_text() == 'PROCESSED: test content'
    # No cleanup needed!
 
def test_config_parser(work_dir):
    # Each test gets a fresh directory
    config = work_dir / 'config.json'
    config.write_text('{"key": "value"}')
    
    result = parse_config(config)
    assert result['key'] == 'value'

Fixture with Pre-populated Files

import tempfile
from pathlib import Path
import pytest
import json
 
@pytest.fixture
def sample_project(work_dir):
    """Create a realistic project structure for testing."""
    (work_dir / 'src').mkdir()
    (work_dir / 'src' / 'main.py').write_text('print("hello")')
    (work_dir / 'tests').mkdir()
    (work_dir / 'tests' / 'test_main.py').write_text('def test_it(): pass')
    (work_dir / 'config.json').write_text(json.dumps({'debug': True}))
    return work_dir
 
def test_project_analyzer(sample_project):
    result = analyze_project(sample_project)
    assert result['has_tests'] is True
    assert result['source_files'] == 1

SpooledTemporaryFile: Best of Both Worlds

For data that might be small or large:

import tempfile
 
# Stays in memory until 5MB, then spills to disk
with tempfile.SpooledTemporaryFile(
    max_size=5 * 1024 * 1024,
    mode='w+b'
) as f:
    f.write(some_data)  # In memory if small
    f.seek(0)
    process(f)

I use this when:

Processing API responses of unknown size
Building data that might be huge but usually isn't
Want memory speed but can't risk OOM

Quick Reference

Function	Use When
`TemporaryFile`	No external access needed
`NamedTemporaryFile`	Need path for other tools
`TemporaryDirectory`	Need scratch directory
`mkstemp`	Low-level file control
`mkdtemp`	Low-level directory control
`SpooledTemporaryFile`	Unknown size, memory-first

The main lesson: stop rolling your own temp file logic. tempfile handles the security, naming, and cleanup correctly. Use it.

React to this post:

#The NamedTemporaryFile delete Gotcha

#The Fix: delete=False

#Python 3.12+ Solution: delete_on_close

#TemporaryFile: When You Don't Need a Path

#TemporaryDirectory: The One I Use Most

#The ignore_cleanup_errors Parameter

#Low-Level Control: mkstemp and mkdtemp

#When I Use Low-Level Functions

#Security: Why tempfile Matters

#1. Race Conditions (TOCTOU)

#2. Predictable Names

#The tempfile Solution

#Where Do Temp Files Go?

#The TMPDIR Environment Variable

#Per-File Override

#Pattern: Processing Uploads

#Pattern: Test Fixtures

#Fixture with Pre-populated Files

#SpooledTemporaryFile: Best of Both Worlds

#Quick Reference

Keep Reading

Need help shipping fast?