{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第9章 学生Notebook：看图盘点助手\n",
    "\n",
    "> 把\"看起来像\"变成证据分级\n",
    "\n",
    "本Notebook完成以下任务：\n",
    "1. 用开放提问和约束提问分别描述图片\n",
    "2. 对模型每句结论做证据分级（A/B/C）\n",
    "3. 用扰动（遮挡/裁剪/降质）检查模型边界\n",
    "4. 修改一个变量，观察变化\n",
    "\n",
    "**准备**：确认 `data/ground_truth.json` 存在。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 加载数据\n",
    "\n",
    "读取图片标注和标准答案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "\n",
    "data_path = Path('../data/ground_truth.json')\n",
    "with open(data_path, 'r', encoding='utf-8') as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "images = data['images']\n",
    "questions = data['questions']\n",
    "\n",
    "print(f'图片数量: {len(images)}')\n",
    "print(f'提问方式: {len(questions)}种')\n",
    "print()\n",
    "for img in images:\n",
    "    n_items = len(img['items'])\n",
    "    print(f'  {img[\"id\"]}: {img[\"description\"]} ({n_items}个已知物品)')\n",
    "print()\n",
    "print(f'开放提问: {questions[\"open\"]}')\n",
    "print(f'约束提问: {questions[\"constrained\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 获取模型描述\n",
    "\n",
    "三种路径：\n",
    "1. 本地视觉模型（Qwen-VL via Ollama）\n",
    "2. 课程云端视觉API\n",
    "3. 预录输出（离线兜底）\n",
    "\n",
    "下面的代码自动检测可用路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# 预录的模型输出（离线兜底）\n",
    "RECORDED_OUTPUTS = {\n",
    "    'desk_normal': {\n",
    "        'open': '图片中有一台笔记本电脑、一本蓝色封面的书、一支铅笔和一个白色杯子。',\n",
    "        'constrained': '1. 笔记本电脑，位于桌面中央偏左。2. 蓝色封面书本，位于右上角。3. 铅笔，位于笔记本右侧。4. 白色杯子，位于右下角。'\n",
    "    },\n",
    "    'desk_occluded': {\n",
    "        'open': '图片中有一台笔记本电脑、一些书和文具。看起来还有一个杯子。',\n",
    "        'constrained': '1. 笔记本电脑，位于桌面中央，可以确认。2. 书本的一部分，右上角露出蓝色封面，但不确定完整书名。3. 疑似有铅笔，被笔记本压住，只能看到一部分。'\n",
    "    },\n",
    "    'desk_dark': {\n",
    "        'open': '图片中有一台发光的笔记本电脑，桌上还有一些物品但不太清楚。可能有一本书和一个杯子。',\n",
    "        'constrained': '1. 笔记本电脑，屏幕发光可以确认。2. 其他区域太暗，无法确认具体物品。'\n",
    "    },\n",
    "    'desk_tricky': {\n",
    "        'open': '图片中有一台笔记本电脑、一个咖啡杯和一支铅笔。',\n",
    "        'constrained': '1. 笔记本电脑，可以确认。2. 咖啡杯——需要确认是真实杯子还是图片中的杯子。3. 铅笔，可以确认。'\n",
    "    },\n",
    "    'desk_cluttered': {\n",
    "        'open': '图片中有很多物品，包括笔记本、手机、耳机、钥匙、文具、水杯、书本、眼镜等。',\n",
    "        'constrained': '桌面上有很多小物品密集排列。可以确认有笔记本电脑和一些文具。其他物品太小或太密集，无法逐一确认。'\n",
    "    },\n",
    "    'desk_empty': {\n",
    "        'open': '图片中有一张木色桌面，上面什么都没有。可能远处有一个小物件。',\n",
    "        'constrained': '桌面基本为空。没有可以确认的物品。'\n",
    "    }\n",
    "}\n",
    "\n",
    "def get_model_description(image_id, question_type):\n",
    "    \"\"\"获取模型描述：尝试本地模型，失败则用预录输出\"\"\"\n",
    "    # 尝试本地视觉模型\n",
    "    try:\n",
    "        import urllib.request\n",
    "        img_path = Path(f'../data/images/{image_id}.png')\n",
    "        if img_path.exists():\n",
    "            import base64\n",
    "            img_data = base64.b64encode(img_path.read_bytes()).decode()\n",
    "            url = 'http://localhost:11434/api/generate'\n",
    "            q = questions[question_type]\n",
    "            payload = json.dumps({\n",
    "                'model': 'qwen2.5-vl:7b',\n",
    "                'prompt': q,\n",
    "                'images': [img_data],\n",
    "                'stream': False\n",
    "            }).encode()\n",
    "            req = urllib.request.Request(url, data=payload, headers={'Content-Type': 'application/json'})\n",
    "            with urllib.request.urlopen(req, timeout=30) as resp:\n",
    "                result = json.loads(resp.read())\n",
    "                return result['response'], 'local_vlm'\n",
    "    except Exception:\n",
    "        pass\n",
    "    # 离线兜底\n",
    "    return RECORDED_OUTPUTS.get(image_id, {}).get(question_type, '无可用输出'), 'recorded'\n",
    "\n",
    "# 测试\n",
    "desc, source = get_model_description('desk_normal', 'open')\n",
    "print(f'模型路径: {source}')\n",
    "print(f'desk_normal 开放提问回答: {desc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 开放提问 vs 约束提问\n",
    "\n",
    "**运行**：对每张图片，分别用开放和约束提问获取描述。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{\"图片\":<20} {\"开放提问\":<40} {\"约束提问\":<40}')\n",
    "print('-' * 100)\n",
    "\n",
    "for img in images:\n",
    "    open_desc, _ = get_model_description(img['id'], 'open')\n",
    "    const_desc, _ = get_model_description(img['id'], 'constrained')\n",
    "    print(f'{img[\"id\"]:<20}')\n",
    "    print(f'  开放: {open_desc[:60]}...' if len(open_desc) > 60 else f'  开放: {open_desc}')\n",
    "    print(f'  约束: {const_desc[:60]}...' if len(const_desc) > 60 else f'  约束: {const_desc}')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**思考**：\n",
    "1. 约束提问是否让模型更谨慎？\n",
    "2. 有没有开放提问中\"看到\"但约束提问中\"不确定\"的物品？\n",
    "3. desk_empty 的开放提问中，模型是否编造了物品？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 证据分级\n",
    "\n",
    "对 desk_normal 和 desk_tricky 的约束提问回答，逐句做证据分级。\n",
    "\n",
    "- **A（画面直接可见）**：能在原图中指出该物品\n",
    "- **B（疑似）**：有局部特征但不能完全确认\n",
    "- **C（无法确认）**：画面中没有足够证据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grade_claims(image_id, model_description, ground_truth_items):\n",
    "    \"\"\"对模型描述中的每个声明做证据分级\"\"\"\n",
    "    gt_names = {item['name'] for item in ground_truth_items}\n",
    "    gt_evidence = {item['name']: item['evidence'] for item in ground_truth_items}\n",
    "    \n",
    "    results = []\n",
    "    # 简单关键词匹配（教学演示用）\n",
    "    for item in ground_truth_items:\n",
    "        name = item['name']\n",
    "        # 检查模型是否提到了这个物品\n",
    "        mentioned = any(keyword in model_description for keyword in name.split('（')[0].split())\n",
    "        if mentioned:\n",
    "            results.append({\n",
    "                'item': name,\n",
    "                'model_mentioned': True,\n",
    "                'ground_truth_evidence': item['evidence'],\n",
    "                'student_grade': ''  # 学生填写\n",
    "            })\n",
    "    return results\n",
    "\n",
    "# desk_normal 证据分级\n",
    "normal_img = images[0]\n",
    "normal_desc, _ = get_model_description('desk_normal', 'constrained')\n",
    "grades = grade_claims('desk_normal', normal_desc, normal_img['items'])\n",
    "\n",
    "print('desk_normal 证据分级:')\n",
    "print(f'{\"物品\":<20} {\"模型提到\":<10} {\"标准答案\":<10} {\"你的分级\":<10}')\n",
    "print('-' * 50)\n",
    "for g in grades:\n",
    "    print(f'{g[\"item\"]:<20} {\"是\" if g[\"model_mentioned\"] else \"否\":<10} {g[\"ground_truth_evidence\"]:<10} {g[\"student_grade\"] or \"待填写\":<10}')\n",
    "\n",
    "print()\n",
    "# desk_tricky 证据分级（注意陷阱）\n",
    "tricky_img = images[3]\n",
    "tricky_desc, _ = get_model_description('desk_tricky', 'constrained')\n",
    "print(f'desk_tricky 约束提问回答: {tricky_desc}')\n",
    "print(f'陷阱: {tricky_img.get(\"trap\", \"无\")}')\n",
    "print('模型是否把打印图片中的咖啡杯当成了真实杯子？')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 扰动测试\n",
    "\n",
    "用程序模拟遮挡、裁剪和低清效果，观察模型输出变化。\n",
    "\n",
    "注意：如果没有真实图片文件，这一步使用预录的扰动输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 预录的扰动输出\n",
    "PERTURBATION_OUTPUTS = {\n",
    "    'desk_normal_cropped': '图片中有一台笔记本电脑的局部，看起来还有书的一角。',\n",
    "    'desk_normal_lowres': '图片中有一台笔记本电脑，其他物品不太清楚。可能有一个杯子。',\n",
    "    'desk_normal_occluded': '图片中有一台笔记本电脑，右上角被遮挡。可以看到铅笔的一部分。'\n",
    "}\n",
    "\n",
    "print('desk_normal 扰动对照:')\n",
    "print()\n",
    "print(f'原始: {RECORDED_OUTPUTS[\"desk_normal\"][\"constrained\"]}')\n",
    "print()\n",
    "for key, output in PERTURBATION_OUTPUTS.items():\n",
    "    label = key.replace('desk_normal_', '')\n",
    "    print(f'{label}: {output}')\n",
    "    print()\n",
    "\n",
    "# 分析哪些结论在扰动后改变了\n",
    "print('扰动影响分析:')\n",
    "print('  裁剪后: 杯子消失了（被裁掉）')\n",
    "print('  低清后: 铅笔变成\"可能\"（细节丢失）')\n",
    "print('  遮挡后: 书和铅笔变成部分可见')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. B档修改\n",
    "\n",
    "从以下两项中选择**一项**：\n",
    "\n",
    "**选项A**：为证据分级新增一条规则\n",
    "- 例如：\"如果模型提到颜色但图片是黑白的，标为C\"\n",
    "- 先预测这条规则会影响哪些结论\n",
    "\n",
    "**选项B**：修改约束提问的措辞\n",
    "- 让提问更严格或更宽松\n",
    "- 先预测模型回答会怎样变化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我选择：选项___\n",
    "# \n",
    "# 修改前预测：\n",
    "# \n",
    "\n",
    "# === 选项A：新增证据分级规则 ===\n",
    "# new_rule = \"如果___，则标为___\"\n",
    "# print(f'新增规则: {new_rule}')\n",
    "# print('受影响的结论:')\n",
    "\n",
    "# === 选项B：修改约束提问 ===\n",
    "# modified_question = questions['constrained'] + ' 额外要求：___'\n",
    "# print(f'修改后的提问: {modified_question}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 验收"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('验收清单:')\n",
    "checks = [\n",
    "    '运行了开放提问和约束提问对照',\n",
    "    '完成了 desk_normal 和 desk_tricky 的证据分级',\n",
    "    '分析了至少一种扰动的影响',\n",
    "    '完成了B档修改',\n",
    "    '能指出模型编造的内容（desk_empty 或 desk_tricky）',\n",
    "]\n",
    "for c in checks:\n",
    "    print(f'  [ ] {c}')\n",
    "\n",
    "print()\n",
    "print('关键概念检查:')\n",
    "print('  1. 视觉模型的输入是___，输出是___')\n",
    "print('  2. 视觉幻觉的常见原因有___')\n",
    "print('  3. 证据分级的三个等级是___')\n",
    "print('  4. 规则模型和视觉模型的分工是___')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}