The os module provides foundational utilities for filesystem interaction. For shallow directory inspection, os.listdir() enumerates immediate children without descending into subfolders.

import os

def scan_shallow(folder_path):
    try:
        entries = os.listdir(folder_path)
        for item in entries:
            full_item_path = os.path.join(folder_path, item)
            print(full_item_path)
    except PermissionError:
        pass

scan_shallow('/var/log')

For recursive descent through directory trees, os.walk() yields a generator producing three-tuples containing the current directory path, subdirectory names, and filenames.

def deep_crawl(root_location):
    for current_root, subdirectories, file_collection in os.walk(root_location):
        for filename in file_collection:
            absolute_file_path = os.path.join(current_root, filename)
            process_file(absolute_file_path)
        
        for dirname in subdirectories:
            full_dir_path = os.path.join(current_root, dirname)
            validate_directory(full_dir_path)

def process_file(path):
    print(f"Processing: {path}")

def validate_directory(path):
    print(f"Entering: {path}")

deep_crawl('/home/user/projects')

Modern Python applications (3.4+) benefit from the pathlib module's object-oriented approach to path manipulation. The Path class encapsulates filesystem operations with in intuitive methods.

from pathlib import Path

def inspect_path_objects(base_directory):
    root = Path(base_directory)
    
    # Non-recursive iteration
    for entry in root.iterdir():
        if entry.is_symlink():
            continue
        elif entry.is_dir():
            print(f"Directory: {entry.name}")
        elif entry.is_file():
            print(f"File: {entry.name}")

def recursive_path_scan(start_point):
    path_obj = Path(start_point)
    for descendant in path_obj.rglob('*'):
        if descendant.is_file():
            print(descendant.resolve())

inspect_path_objects('/tmp')
recursive_path_scan('/etc')

Pattern matching against filenames utilizes the glob module, which supports Unix shell-style wildcards including asterisks and question marks. Recursive globbing requires the ** pattern with recursive=True (Python 3.5+).

import glob

def pattern_match_search(start_dir, file_pattern):
    search_path = os.path.join(start_dir, '**', file_pattern)
    matches = glob.glob(search_path, recursive=True)
    return matches

def specific_extension_hunt(directory, extension):
    pattern = f"*.{extension.lstrip('.')}"
    target = os.path.join(directory, pattern)
    return glob.glob(target)

log_files = pattern_match_search('/var', '*.log')
pdfs = specific_extension_hunt('/home/user', 'pdf')

Selecting a appropriate traversal strategy depends on specific operational constraints. Consider these dimensions when implementing directory walks:

Recursion Requirements: Use os.walk() or Path.rglob() when descending into nested hierarchies. For single-level scans, prefer os.listdir() or Path.iterdir() to minimize overhead.

Path Manipulation Complexity: pathlib reduces string concatenation errors through operator overloading and methods like joinpath(), parent, and suffix. Traditional string-based path joining with os.path.join() requires manual separator management.

Filtering Capabilities: When searching by filename patterns (e.g., *.config, data_??.json), glob provides built-in wildcard support without explicit conditional logic. Standard loops require manual string checking via endswith() or regular expressions.

Version Compatibility: pathlib requires Python 3.4 or newer. Codebases supporting Python 2.7 or early 3.x versions must rely on os and glob modules exclusively.

Memory and Performance: os.walk() and generators process directories iteratively, maintaining constant memory regardless of tree size. Materializing complete file lists (e.g., glob.glob() without iteration) consumes memory proportional to result set size. For massive filesystems, consider asynchronous I/O or multiprocessing to parallelize traversal across subtrees.

The following example demonstrates aggregating statistics for specific file types across a complex directory structure:

import os
from collections import defaultdict

def audit_file_inventory(search_root, target_extensions):
    statistics = {
        'count': 0,
        'total_bytes': 0,
        'largest_file': None,
        'max_size': 0
    }
    
    extension_set = {ext.lower().lstrip('.') for ext in target_extensions}
    
    for dirpath, dirnames, filenames in os.walk(search_root):
        # Skip hidden directories
        dirnames[:] = [d for d in dirnames if not d.startswith('.')]
        
        for filename in filenames:
            file_ext = filename.split('.')[-1].lower() if '.' in filename else ''
            
            if file_ext in extension_set:
                complete_path = os.path.join(dirpath, filename)
                
                try:
                    file_stats = os.stat(complete_path)
                    file_size = file_stats.st_size
                    
                    statistics['count'] += 1
                    statistics['total_bytes'] += file_size
                    
                    if file_size > statistics['max_size']:
                        statistics['max_size'] = file_size
                        statistics['largest_file'] = complete_path
                        
                except (OSError, IOError):
                    continue
    
    return statistics

results = audit_file_inventory('/data/archive', ['.txt', '.md', '.rst'])
print(f"Found {results['count']} documents totaling {results['total_bytes']} bytes")
if results['largest_file']:
    print(f"Largest document: {results['largest_file']} ({results['max_size']} bytes)")