MERRA-2 tavgM_2d_aer_Nx 月尺度数据Python 自动化下载与预处理流程在气象与环境研究领域MERRA-2Modern-Era Retrospective analysis for Research and Applications, Version 2数据集因其高时空分辨率和长期连续性成为气溶胶、大气成分等研究的重要数据源。其中tavgM_2d_aer_Nx作为月平均气溶胶产品广泛应用于区域空气质量分析、气候效应评估等场景。传统基于wget的命令行下载方式虽然简单直接但在批量下载、错误处理、数据预处理等方面存在明显局限。本文将展示如何用Python构建一套完整的自动化工作流从认证、查询到下载、预处理一气呵成。1. 环境配置与认证机制1.1 必备库安装处理MERRA-2数据需要以下Python库支持pip install requests netCDF4 numpy pandas tqdm核心功能说明requests处理HTTP请求与认证netCDF4读取NetCDF格式数据numpy/pandas数据预处理tqdm下载进度显示1.2 NASA Earthdata认证NASA GES DISC采用统一认证系统需提前注册账号并获取授权import requests from netrc import netrc # 建议将认证信息存储在~/.netrc文件 auth netrc().authenticators(urs.earthdata.nasa.gov) session requests.Session() session.auth (auth[0], auth[2]) # (username, password)注意避免在代码中硬编码密码推荐使用环境变量或配置文件管理敏感信息2. 数据检索与元信息获取2.1 产品目录查询通过API获取tavgM_2d_aer_Nx产品元数据import json product_url https://disc.gsfc.nasa.gov/service/datasets/json/M2TMNXAER_5.12.4 response session.get(product_url) metadata json.loads(response.text) # 提取关键信息 variables metadata[Variables] time_range metadata[Temporal Coverage] spatial_res metadata[Spatial Resolution]2.2 变量速查表生成将变量信息转换为结构化表格变量名描述单位有效范围TOTEXTTAU总消光光学厚度10-5DUSTEXTTAU沙尘消光光学厚度10-3............3. 自动化下载实现3.1 分块下载函数实现带重试机制的下载功能from tqdm import tqdm import os def download_file(url, save_path, max_retries3): for attempt in range(max_retries): try: with session.get(url, streamTrue) as r: r.raise_for_status() total_size int(r.headers.get(content-length, 0)) with open(save_path, wb) as f, tqdm( unitB, unit_scaleTrue, totaltotal_size ) as pbar: for chunk in r.iter_content(chunk_size8192): f.write(chunk) pbar.update(len(chunk)) return True except Exception as e: print(fAttempt {attempt1} failed: {str(e)}) return False3.2 时间范围批量下载生成指定时间段的下载链接from datetime import datetime, timedelta def generate_download_links(start_date, end_date, base_url): current start_date while current end_date: year_month current.strftime(%Y%m) url f{base_url}/MERRA2_400.tavgM_2d_aer_Nx.{year_month}.nc4 yield url current timedelta(days32) current current.replace(day1)4. 数据预处理与质量检查4.1 快速数据探查使用netCDF4进行初步检查from netCDF4 import Dataset def inspect_nc_file(filepath): with Dataset(filepath) as nc: print(Dimensions:, nc.dimensions.keys()) print(Variables:, nc.variables.keys()) # 提取时间维度信息 time_var nc.variables[time] print(fTime range: {time_var[0]} to {time_var[-1]})4.2 数据裁剪与重采样针对区域研究的需求import xarray as xr def regional_subset(filepath, lat_range, lon_range): ds xr.open_dataset(filepath) return ds.sel( latslice(*lat_range), lonslice(*lon_range) )5. 错误处理与日志记录5.1 异常处理机制完善下载流程的健壮性import logging logging.basicConfig(filenamemerra2_download.log, levellogging.INFO) def safe_download(url, save_dir): try: filename url.split(/)[-1] save_path os.path.join(save_dir, filename) if os.path.exists(save_path): logging.info(fFile exists: {filename}) return True success download_file(url, save_path) if success: logging.info(fDownloaded: {filename}) return success except Exception as e: logging.error(fFailed {url}: {str(e)}) return False5.2 断点续传实现记录下载进度状态import pickle class DownloadTracker: def __init__(self, state_filedownload_state.pkl): self.state_file state_file self.completed set() self.load_state() def load_state(self): if os.path.exists(self.state_file): with open(self.state_file, rb) as f: self.completed pickle.load(f) def save_state(self): with open(self.state_file, wb) as f: pickle.dump(self.completed, f) def mark_complete(self, url): self.completed.add(url) self.save_state()6. 完整工作流示例整合各模块的端到端流程def merra2_pipeline(start_date, end_date, save_dir, regionNone): os.makedirs(save_dir, exist_okTrue) tracker DownloadTracker() base_url https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2/M2TMNXAER.5.12.4 for url in generate_download_links(start_date, end_date, base_url): if url in tracker.completed: continue if safe_download(url, save_dir): filepath os.path.join(save_dir, url.split(/)[-1]) # 数据质量检查 try: inspect_nc_file(filepath) if region: ds regional_subset(filepath, *region) output_path filepath.replace(.nc4, _subset.nc4) ds.to_netcdf(output_path) tracker.mark_complete(url) except Exception as e: logging.error(fProcessing failed {url}: {str(e)}) os.remove(filepath)实际项目中这套工作流帮助我高效处理了2010-2020年的东亚区域气溶胶数据相比传统手动下载方式节省约80%的时间成本。关键点在于合理设置并发下载数NASA服务器限制为2-3个并发连接和做好本地状态管理避免重复下载。