PaliGemma2 目标检测微调实战用自定义数据集训练扑克牌检测模型这篇教程是我根据 PaliGemma2 多模态目标检测微调、自定义数据集导出和 supervision 评估流程整理出来的。重点演示如何把 从数据集后台导出的 PaliGemma JSONL 数据集用于检测任务微调并用 mAP、混淆矩阵和可视化样例评估结果。PaliGemma2 可以把目标检测任务表述为“图像 文本前缀 - 文本后缀”的生成式任务。这里的 suffix 保存了类别和框坐标训练完成后再用supervision解析为标准检测结果。本文会重点跑通以下流程下载 PaliGemma 格式目标检测数据集用 JSONL 样本构造图像和文本训练对冻结视觉编码器进行轻量微调用supervision解析 PaliGemma 检测文本计算 mAP 和混淆矩阵评估模型如果你正在系统学习多模态微调、目标检测、OCR 或图像分割建议收藏本文配套 notebook、示例图片和运行环境说明后续会继续整理。如果环境配置卡住可以在评论区说明具体报错。 文章目录PaliGemma2 目标检测微调实战用自定义数据集训练扑克牌检测模型⚙️ 环境准备 下载 PaliGemma 数据集 加载并预览 JSONL 数据 加载 PaliGemma2 模型️ 微调 PaliGemma2 检测模型 使用微调模型推理 评估检测效果 小结 同系列教程汇总⚙️ 环境准备检查 GPU安装 supervision、PEFT、bitsandbytes 和指定版本 transformers。!nvidia-smi 下载 PaliGemma 数据集从数据集后台获取扑克牌检测数据集导出格式为paligemma。!pip install-q supervision peft bitsandbytes transformers4.47.0fromtypesimportSimpleNamespace# 从数据集后台下载并解压数据集后修改 DATASET_DIR 指向数据集目录。DATASET_DIR/content/dataset# 修改为数据集后台导出的数据集目录datasetSimpleNamespace(locationDATASET_DIR)!head-n5{dataset.location}/dataset/_annotations.train.jsonl 加载并预览 JSONL 数据定义 JSONL 数据集类读取图片和标注文本并用 supervision 预览训练样本。importosimportjsonimportrandomfromPILimportImagefromtorch.utils.dataimportDatasetclassJSONLDataset(Dataset):def__init__(self,jsonl_file_path:str,image_directory_path:str):self.jsonl_file_pathjsonl_file_path self.image_directory_pathimage_directory_path self.entriesself._load_entries()def_load_entries(self):entries[]withopen(self.jsonl_file_path,r)asfile:forlineinfile:datajson.loads(line)entries.append(data)returnentriesdef__len__(self):returnlen(self.entries)def__getitem__(self,idx:int):ifidx0oridxlen(self.entries):raiseIndexError(Index out of range)entryself.entries[idx]image_pathos.path.join(self.image_directory_path,entry[image])imageImage.open(image_path)returnimage,entrytrain_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/dataset/_annotations.train.jsonl,image_directory_pathf{dataset.location}/dataset,)valid_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/dataset/_annotations.valid.jsonl,image_directory_pathf{dataset.location}/dataset,)test_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/dataset/_annotations.test.jsonl,image_directory_pathf{dataset.location}/dataset,)fromtqdmimporttqdmimportsupervisionassv CLASSEStrain_dataset[0][1][prefix].replace(detect ,).split( ; )images[]foriinrange(25):image,labeltrain_dataset[i]detectionssv.Detections.from_lmm(lmmpaligemma,resultlabel[suffix],resolution_wh(image.width,image.height),classesCLASSES)imagesv.BoxAnnotator(thickness4).annotate(image,detections)imagesv.LabelAnnotator(text_scale2,text_thickness4).annotate(image,detections)images.append(image)sv.plot_images_grid(images,(5,5)) 加载 PaliGemma2 模型加载 PaliGemma2 processor 和模型并提供冻结视觉编码器或 LoRA/QLoRA 两种训练方式。importtorchfromtransformersimportPaliGemmaProcessor,PaliGemmaForConditionalGeneration MODEL_IDgoogle/paligemma2-3b-pt-448DEVICEtorch.device(cudaiftorch.cuda.is_available()elsecpu)fromhuggingface_hubimportnotebook_login notebook_login()processorPaliGemmaProcessor.from_pretrained(MODEL_ID)# title Freeze the image encoderTORCH_DTYPEtorch.bfloat16 modelPaliGemmaForConditionalGeneration.from_pretrained(MODEL_ID,torch_dtypeTORCH_DTYPE).to(DEVICE)forparaminmodel.vision_tower.parameters():param.requires_gradFalseforparaminmodel.multi_modal_projector.parameters():param.requires_gradFalse# title Fine-tune the entire model with LoRA and QLoRA# from transformers import BitsAndBytesConfig# from peft import get_peft_model, LoraConfig# bnb_config BitsAndBytesConfig(# load_in_4bitTrue,# bnb_4bit_compute_dtypetorch.bfloat16# )# lora_config LoraConfig(# r8,# target_modules[q_proj, o_proj, k_proj, v_proj, gate_proj, up_proj, down_proj],# task_typeCAUSAL_LM,# )# model PaliGemmaForConditionalGeneration.from_pretrained(MODEL_ID, device_mapauto)# model get_peft_model(model, lora_config)# model.print_trainable_parameters()# TORCH_DTYPE model.dtype️ 微调 PaliGemma2 检测模型构造 collate 函数将检测文本后缀作为训练目标并使用 Transformers Trainer 启动训练。fromtransformersimportTrainer,TrainingArgumentsdefaugment_suffix(suffix):partssuffix.split( ; )random.shuffle(parts)return ; .join(parts)defcollate_fn(batch):images,labelszip(*batch)paths[label[image]forlabelinlabels]prefixes[imagelabel[prefix]forlabelinlabels]suffixes[augment_suffix(label[suffix])forlabelinlabels]inputsprocessor(textprefixes,imagesimages,return_tensorspt,suffixsuffixes,paddinglongest).to(TORCH_DTYPE).to(DEVICE)returninputs argsTrainingArguments(num_train_epochs16,remove_unused_columnsFalse,per_device_train_batch_size1,gradient_accumulation_steps16,warmup_steps2,learning_rate2e-5,weight_decay1e-6,adam_beta20.999,logging_steps50,optimadamw_hf,save_strategysteps,save_steps1000,save_total_limit1,output_dirpaligemma2_object_detection,bf16True,report_to[tensorboard],dataloader_pin_memoryFalse)trainerTrainer(modelmodel,train_datasettrain_dataset,eval_datasetvalid_dataset,data_collatorcollate_fn,argsargs)trainer.train() 使用微调模型推理在测试集上生成检测文本解析为检测框并可视化。image,labeltest_dataset[1]prefiximagelabel[prefix]suffixlabel[suffix]inputsprocessor(textprefix,imagesimage,return_tensorspt).to(TORCH_DTYPE).to(DEVICE)prefix_lengthinputs[input_ids].shape[-1]withtorch.inference_mode():generationmodel.generate(**inputs,max_new_tokens256,do_sampleFalse)generationgeneration[0][prefix_length:]decodedprocessor.decode(generation,skip_special_tokensTrue)print(decoded)w,himage.size detectionssv.Detections.from_lmm(lmmpaligemma,resultdecoded,resolution_wh(w,h),classesCLASSES)annotated_imageimage.copy()annotated_imagesv.BoxAnnotator().annotate(annotated_image,detections)annotated_imagesv.LabelAnnotator(smart_positionTrue).annotate(annotated_image,detections)annotated_image 评估检测效果遍历测试集生成预测计算 mAP、混淆矩阵并绘制预测样例网格。importnumpyasnpfromtqdmimporttqdm images[]targets[]predictions[]withtorch.inference_mode():foriintqdm(range(len(test_dataset))):image,labeltest_dataset[i]prefiximagelabel[prefix]suffixlabel[suffix]inputsprocessor(textprefix,imagesimage,return_tensorspt).to(TORCH_DTYPE).to(DEVICE)prefix_lengthinputs[input_ids].shape[-1]generationmodel.generate(**inputs,max_new_tokens256,do_sampleFalse)generationgeneration[0][prefix_length:]generated_textprocessor.decode(generation,skip_special_tokensTrue)w,himage.size predictionsv.Detections.from_lmm(lmmpaligemma,resultgenerated_text,resolution_wh(w,h),classesCLASSES)prediction.class_idnp.array([CLASSES.index(class_name)forclass_nameinprediction[class_name]])prediction.confidencenp.ones(len(prediction))targetsv.Detections.from_lmm(lmmpaligemma,resultsuffix,resolution_wh(w,h),classesCLASSES)target.class_idnp.array([CLASSES.index(class_name)forclass_nameintarget[class_name]])images.append(image)targets.append(target)predictions.append(prediction)# title Calculate mAPfromsupervision.metricsimportMeanAveragePrecision,MetricTarget map_metricMeanAveragePrecision(metric_targetMetricTarget.BOXES)map_resultmap_metric.update(predictions,targets).compute()print(map_result)map_result.plot()# title Calculate Confusion Matrixconfusion_matrixsv.ConfusionMatrix.from_detections(predictionspredictions,targetstargets,classesCLASSES)_confusion_matrix.plot()annotated_images[]foriinrange(25):imageimages[i]detectionspredictions[i]annotated_imageimage.copy()annotated_imagesv.BoxAnnotator(thickness4).annotate(annotated_image,detections)annotated_imagesv.LabelAnnotator(text_scale2,text_thickness4,smart_positionTrue).annotate(annotated_image,detections)annotated_images.append(annotated_image)sv.plot_images_grid(annotated_images,(5,5)) 小结这篇教程完整整理了Fine-Tune PaliGemma2 on Object Detection Dataset的核心复现流程。实际操作时建议先确认 GPU、依赖版本、数据集路径和模型权限再逐段运行 notebook。下载 PaliGemma 格式目标检测数据集用 JSONL 样本构造图像和文本训练对冻结视觉编码器进行轻量微调用supervision解析 PaliGemma 检测文本计算 mAP 和混淆矩阵评估模型后续我会继续按源项目顺序整理 项目教程 中的目标检测、实例分割、OCR、多目标跟踪和视觉大模型教程。 同系列教程汇总Google Gemini 3.5 Flash 零样本目标检测教程从提示词到可视化结果GLM-OCR 文档识别实战教程从验证码、公式到车牌 OCRRF-DETR ByteTrack 多目标跟踪实战教程从命令行到 Python 视频轨迹可视化SAM 3 图像分割实战教程文本、框和点提示的多种分割方式PaliGemma2 目标检测微调实战用自定义数据集训练扑克牌检测模型-本文