# Codebase Structure

**Analysis Date:** 2026-03-21

## Directory Layout

```
JobData/                          # Project root
├── app/                          # FastAPI backend (Python)
│   ├── __init__.py               # App factory create_app()
│   ├── api/
│   │   └── v1/                   # All REST API routes (/api/v1/...)
│   │       ├── __init__.py       # Router aggregation + permission guards
│   │       ├── analytics.py      # Data analytics endpoints
│   │       ├── stats.py          # Table count stats endpoints
│   │       ├── pipeline.py       # ECS pipeline trigger endpoint
│   │       ├── base/             # Login, user-info, menu-tree (no auth)
│   │       ├── job/              # Data ingest endpoints (batch/single/async)
│   │       ├── cleaning/         # Targeted cleaning endpoints
│   │       ├── company/          # Company search endpoints
│   │       ├── keyword/          # Keyword CRUD endpoints
│   │       ├── token/            # Boss token management endpoints
│   │       ├── proxy/            # Proxy IP management endpoints
│   │       ├── users/            # User CRUD
│   │       ├── roles/            # Role + permission assignment
│   │       ├── menus/            # Menu tree CRUD
│   │       ├── apis/             # API registration management
│   │       ├── depts/            # Department tree CRUD
│   │       └── auditlog/         # Audit log query
│   ├── controllers/              # CRUD orchestration for MySQL entities
│   │   ├── api.py                # ApiController (refresh_api(), scan routes)
│   │   ├── user.py               # UserController
│   │   ├── role.py               # RoleController
│   │   ├── menu.py               # MenuController
│   │   ├── dept.py               # DeptController
│   │   ├── keyword.py            # KeywordController
│   │   ├── proxy.py              # ProxyController
│   │   ├── token.py              # TokenController
│   │   ├── cleaning.py           # CleaningController
│   │   └── company.py            # CompanyController (search via crawler services)
│   ├── core/                     # Framework infrastructure
│   │   ├── init_app.py           # lifespan: migrations, seed, ClickHouse init
│   │   ├── scheduler.py          # APScheduler job definitions + registration
│   │   ├── clickhouse.py         # ClickHouseManager singleton
│   │   ├── clickhouse_init.py    # ClickHouse DDL (tables + analytics view)
│   │   ├── dependency.py         # AuthControl, PermissionControl, DependPermission
│   │   ├── locks.py              # DistributedLock (Redis + file fallback)
│   │   ├── middlewares.py        # BackGroundTaskMiddleware, HttpAuditLogMiddleware
│   │   ├── ip_tracking.py        # IpTrackingMiddleware
│   │   ├── exceptions.py         # Custom exception classes + handlers
│   │   ├── crud.py               # Generic ListParams, pagination helpers
│   │   ├── ctx.py                # CTX_USER_ID ContextVar
│   │   ├── bgtask.py             # BgTasks helper for BackGroundTaskMiddleware
│   │   ├── proxy_rule.py         # Proxy selection rules
│   │   └── algorithms/
│   │       └── antispider.py     # Anti-spider signature helpers
│   ├── models/                   # Tortoise-ORM models (MySQL)
│   │   ├── base.py               # BaseModel, TimestampMixin, UUIDModel
│   │   ├── admin.py              # User, Role, Api, Menu, Dept, AuditLog
│   │   ├── token.py              # BossToken
│   │   ├── cleaning.py           # CleaningTask / CleaningRecord
│   │   ├── metrics.py            # ScheduledTaskRun, StatsTotal, IpUploadStats
│   │   ├── company.py            # CompanyCleaningQueue
│   │   ├── keyword.py            # BossKeyword, QcwyKeyword, ZhilianKeyword
│   │   └── enums.py              # MethodType enum
│   ├── repositories/             # ClickHouse query layer
│   │   └── clickhouse_repo.py    # ClickHouseBaseRepo, JobAnalyticsRepo
│   ├── schemas/                  # Pydantic request/response schemas
│   │   ├── ingest.py             # IngestBatchRequest, PlatformType, DataType enums
│   │   ├── analytics.py          # Analytics query params + response schemas
│   │   ├── keyword.py            # Keyword schemas
│   │   ├── token.py              # Token schemas
│   │   ├── base.py               # Common response envelopes
│   │   ├── login.py              # Login request/response
│   │   ├── users.py              # User schemas
│   │   ├── roles.py              # Role schemas
│   │   ├── menus.py              # Menu schemas (MenuType enum)
│   │   ├── apis.py               # API schemas
│   │   ├── depts.py              # Dept schemas
│   │   ├── cleaning.py           # Cleaning schemas
│   │   └── proxy.py              # Proxy schemas
│   ├── services/                 # Business logic
│   │   ├── analytics_service.py  # AnalyticsService → JobAnalyticsRepo
│   │   ├── cleaning.py           # CleaningService (targeted cleaning)
│   │   ├── company_cleaner.py    # CompanyCleaner (automated company pipeline)
│   │   ├── company_jobs_sync.py  # CompanyJobsSyncService
│   │   ├── company_storage.py    # company_storage (ClickHouse company insert)
│   │   ├── ingest/               # Ingest pipeline subsystem
│   │   │   ├── service.py        # IngestService.store_batch/store_single/query_data
│   │   │   ├── registry.py       # PlatformConfig registry + get_config()
│   │   │   ├── dedup.py          # batch_dedup_filter() against ClickHouse
│   │   │   ├── remote_push.py    # Async HTTP push to secondary target
│   │   │   └── configs/          # Per-platform PlatformConfig registrations
│   │   └── crawler/              # In-process platform HTTP clients
│   │       ├── _base.py          # BaseFetcher, BaseSearcher, ApiResult
│   │       ├── _http_client.py   # HTTPClient wrapper (sync requests)
│   │       ├── _boss_client.py   # Boss HTTP client
│   │       ├── _boss_api.py      # Boss API methods
│   │       ├── _boss_sign.py     # Boss request signing
│   │       ├── _zhilian_client.py
│   │       ├── _zhilian_api.py
│   │       ├── _zhilian_sign.py
│   │       ├── _job51_client.py
│   │       ├── _job51_api.py
│   │       ├── _job51_sign.py
│   │       ├── boss.py           # BossService (high-level facade)
│   │       ├── qcwy.py           # QcwyService (high-level facade)
│   │       └── zhilian.py        # ZhilianService (high-level facade)
│   ├── settings/
│   │   └── config.py             # Settings (pydantic-settings BaseSettings)
│   ├── log/
│   │   └── __init__.py           # loguru logger export
│   └── utils/                    # Shared utilities
├── web/                          # Vue 3 frontend
│   ├── src/
│   │   ├── main.js               # App entry, Pinia + Router setup
│   │   ├── App.vue               # Root component
│   │   ├── api/                  # Axios API call modules
│   │   │   ├── index.js          # System management APIs
│   │   │   ├── analytics.js      # Analytics APIs
│   │   │   ├── keyword.js        # Keyword APIs
│   │   │   ├── proxy.js          # Proxy APIs
│   │   │   └── token.js          # Token APIs
│   │   ├── components/           # Shared components (CrudTable, CrudModal, QueryBar)
│   │   ├── layout/               # App shell (sidebar, topbar, tabs)
│   │   ├── router/               # Vue Router config + dynamic route loading
│   │   │   ├── index.js          # createRouter(), addDynamicRoutes()
│   │   │   ├── guard/            # Navigation guards (auth, loading, title)
│   │   │   └── routes/           # Static route definitions
│   │   ├── store/                # Pinia stores
│   │   │   └── modules/          # user, permission, app, tags
│   │   ├── utils/
│   │   │   ├── auth/             # JWT token read/write
│   │   │   ├── http/             # Axios instance + interceptors
│   │   │   └── storage/          # localStorage wrapper
│   │   └── views/                # Page-level Vue components
│   │       ├── login/            # Login page
│   │       ├── analytics/        # ECharts dashboard (trend + distribution)
│   │       ├── recruitment/      # Job data browser (boss/, qcwy/, zhilian/)
│   │       ├── cleaning/         # Cleaning operation + monitor pages
│   │       ├── keyword/          # Keyword management
│   │       ├── profile/          # User profile
│   │       └── system/           # RBAC admin (user, role, menu, api, dept, auditlog, proxy, token)
│   ├── vite.config.js            # Vite build config (plugins, proxy)
│   └── package.json              # Frontend dependencies
├── jobs_spider/                  # Standalone crawlers (external process)
│   ├── boss/
│   │   ├── boos_api.py           # BossZhipinAPI core + SmartIPManager + main (~2246 lines)
│   │   ├── company_spider.py     # Boss company batch spider
│   │   └── enumerate_combos.py   # City × keyword combo enumeration
│   ├── qcwy/
│   │   ├── qcwy.py               # QCWY API wrapper
│   │   ├── search_company_jobs.py
│   │   └── run_company_search.py
│   └── zhilian/
│       ├── zhilian_single.py     # Zhilian single-job spider
│       └── company_spider.py     # Zhilian company spider
├── spiderJobs/                   # Second-generation spider framework (structured)
│   ├── core/
│   │   ├── base.py               # Shared base classes
│   │   └── http_client.py        # HTTP client abstraction
│   ├── platforms/                # Platform-specific implementations
│   │   ├── boss/
│   │   ├── job51/
│   │   └── zhilian/
│   └── runner/                   # Execution runners
│       ├── api_client.py         # Backend API push client
│       ├── loop.py               # Job crawl loop runner
│       └── company_loop.py       # Company crawl loop runner
├── migrations/
│   └── models/                   # Aerich migration files (auto-generated)
├── static/                       # Served at /static/ (built frontend or assets)
├── deploy/                       # Deployment configs and sample images
├── docs/                         # Project documentation
├── ecs_full_pipeline.py          # Alibaba Cloud ECS batch provisioning script
├── create_ecs_instances.py       # ECS instance creation utilities
├── reclean_qcwy_jobs.py          # Standalone QCWY re-cleaning script
├── run.py                        # Uvicorn launch entry point
├── Pipfile / Pipfile.lock        # Python dependency management (pipenv)
├── pyproject.toml                # ruff + black + isort tool config
└── Dockerfile                    # Container build spec
```

## Directory Purposes

**`app/api/v1/`:**
- Purpose: One sub-package per API domain; each contains `__init__.py` exporting its `APIRouter`
- Contains: Route handler functions, dependency injection wiring, Pydantic schema imports
- Key files: `app/api/v1/__init__.py` (aggregation), `app/api/v1/job/job.py` (ingest routes), `app/api/v1/analytics.py`

**`app/controllers/`:**
- Purpose: Business-operation classes for MySQL-backed entities; encapsulate Tortoise-ORM queries
- Contains: One controller per domain entity; typically expose `create`, `update`, `delete`, `list`, `get` methods
- Key files: `app/controllers/api.py` (scan and register FastAPI routes → DB)

**`app/services/ingest/`:**
- Purpose: Self-contained data pipeline: registry lookup → dedup check → ClickHouse insert → remote push
- Contains: `service.py` (main orchestrator), `registry.py` (config store), `dedup.py`, `remote_push.py`, `configs/` (registrations per platform)
- Key files: `app/services/ingest/service.py`, `app/services/ingest/registry.py`

**`app/services/crawler/`:**
- Purpose: In-process HTTP clients for platform APIs, used by backend scheduler (company cleaning) and company search controller
- Contains: Underscore-prefixed private modules (`_boss_client.py`, `_boss_sign.py`, etc.) and public service facades (`boss.py`, `qcwy.py`, `zhilian.py`)
- Key files: `app/services/crawler/_base.py` (shared base classes)

**`app/core/`:**
- Purpose: Framework plumbing — startup, middleware chain, auth, distributed locking, ClickHouse connection
- Key files: `app/core/dependency.py` (auth guards), `app/core/scheduler.py` (all cron jobs), `app/core/locks.py`

**`app/models/`:**
- Purpose: All Tortoise-ORM model definitions; migrated via Aerich (not ClickHouse)
- Key files: `app/models/admin.py` (RBAC), `app/models/keyword.py` (crawl state tracking)

**`app/repositories/`:**
- Purpose: Typed ClickHouse query encapsulation following the Repository pattern
- Key files: `app/repositories/clickhouse_repo.py`

**`jobs_spider/`:**
- Purpose: Standalone scripts that run on ECS nodes; communicate with backend only via HTTP
- Key files: `jobs_spider/boss/boos_api.py` (2246-line core with SmartIPManager, BossZhipinAPI)

**`spiderJobs/`:**
- Purpose: Newer modular spider framework; parallel architecture to `jobs_spider/` with cleaner separation
- Key files: `spiderJobs/runner/api_client.py`, `spiderJobs/runner/loop.py`

**`web/src/views/`:**
- Purpose: One directory per route group; each view owns its local route definition in `route.js`
- Key files: `web/src/views/analytics/index.vue` (ECharts dashboard), `web/src/views/cleaning/monitor.vue`

## Key File Locations

**Entry Points:**
- `run.py`: Uvicorn startup; reads env vars `APP_HOST`, `APP_PORT`, `UVICORN_WORKERS`
- `app/__init__.py`: FastAPI app factory `create_app()` and lifespan context manager
- `web/src/main.js`: Vue 3 app entry; mounts Pinia, Router, directives

**Configuration:**
- `app/settings/config.py`: All backend config (`Settings` extends `pydantic_settings.BaseSettings`); override via env vars
- `web/vite.config.js`: Vite build and dev server config
- `pyproject.toml`: ruff, black, isort tool configuration

**Core Logic:**
- `app/core/init_app.py`: Startup sequence (migrations, seed data, ClickHouse init)
- `app/core/scheduler.py`: All APScheduler job functions and `start_scheduler()`
- `app/core/clickhouse_init.py`: All ClickHouse `CREATE TABLE IF NOT EXISTS` DDL statements
- `app/services/ingest/registry.py`: `PlatformConfig` dataclass + global registry dict
- `app/services/ingest/service.py`: `IngestService.store_batch()` — main data ingestion path
- `app/services/company_cleaner.py`: `CompanyCleaner` — automated company data pipeline

**ClickHouse Schema:**
- `app/core/clickhouse_init.py`: Defines all 6 MergeTree tables and 1 analytics VIEW DDL; apply via `ClickHouseInitializer.initialize_all_tables()`

**MySQL Migrations:**
- `migrations/models/`: Auto-generated by `aerich migrate`; applied via `aerich upgrade` or automatically on startup

**Testing:**
- `tests/test_company_jobs_sync.py`
- `tests/test_company_storage.py`

## Naming Conventions

**Files (Python):**
- Modules: `snake_case.py` (e.g., `company_cleaner.py`, `clickhouse_init.py`)
- Private crawler modules: leading underscore prefix (e.g., `_boss_client.py`, `_zhilian_sign.py`)
- Spider scripts in `jobs_spider/`: descriptive names without prefix (e.g., `boos_api.py`, `zhilian_single.py`)

**Files (Vue/JS):**
- Pages: `index.vue` for primary view, `monitor.vue` / descriptive name for secondary views
- API modules: `snake_case.js` matching domain (e.g., `analytics.js`, `keyword.js`)
- Router guard files: Grouped in `router/guard/` directory

**Classes (Python):**
- Services: `PascalCaseService` (e.g., `IngestService`, `AnalyticsService`, `CompanyCleaner`)
- Controllers: `PascalCaseController` with module-level singleton instance in snake_case (e.g., `user_controller = UserController()`)
- Repos: `PascalCaseRepo` (e.g., `ClickHouseBaseRepo`, `JobAnalyticsRepo`)
- Models: `PascalCase` matching table concept (e.g., `BossToken`, `CompanyCleaningQueue`)

**Directories:**
- Domain groupings: lowercase (e.g., `cleaning/`, `recruitment/`, `keyword/`)
- Platform names used consistently: `boss`, `qcwy`, `zhilian` (lowercase) across all layers

## Where to Add New Code

**New API Endpoint:**
1. Create or update router file in `app/api/v1/{domain}/`
2. Register the router in `app/api/v1/__init__.py` with `v1_router.include_router(...)`
3. Run `api_controller.refresh_api()` on next app startup to sync new routes to the `api` table
4. Add corresponding Pydantic schemas in `app/schemas/{domain}.py`

**New Platform Data Type (Ingest):**
1. Define `PlatformConfig` with dedup fields in `app/services/ingest/configs/`
2. Call `register(config)` at module import in `app/services/ingest/configs/__init__.py`
3. ClickHouse table DDL goes in `app/core/clickhouse_init.py` under `_TABLE_DDLS`

**New MySQL Model:**
1. Add model class to appropriate file in `app/models/` (or create new file)
2. Register the model module in `app/settings/config.py` under `TORTOISE_ORM.apps.models.models`
3. Run `aerich migrate && aerich upgrade` (or rely on `RUN_MIGRATIONS_ON_STARTUP=True`)

**New Scheduled Job:**
1. Define async job function in `app/core/scheduler.py`
2. Wrap execution body with `DistributedLock` context from `app/core/locks.py`
3. Add `scheduler.add_job(...)` call inside `start_scheduler()`
4. Record run result via `_record_task_run()` to `ScheduledTaskRun` table

**New Frontend Page:**
1. Create view component at `web/src/views/{module}/index.vue`
2. Add `route.js` in the same directory with route meta
3. Register the route in `web/src/router/routes/` static routes or rely on backend menu system
4. Add corresponding backend menu entry in `app/core/init_app.py` → `init_menus()` (for seed data)

**New In-Process Platform Crawler:**
1. Add private client modules `_platform_client.py`, `_platform_api.py`, `_platform_sign.py` under `app/services/crawler/`
2. Create public facade `platform.py` extending or wrapping `BaseFetcher` / `BaseSearcher` from `app/services/crawler/_base.py`
3. Instantiate in `CompanyCleaner.__init__()` if needed for company pipeline

**Shared Utilities:**
- Backend Python utilities: `app/utils/`
- Frontend JS utilities: `web/src/utils/`

## Special Directories

**`.startup_lock/`:**
- Purpose: File-based mutex directory used to prevent multi-worker race on startup seed data
- Generated: Yes (created and deleted at startup)
- Committed: No (in `.gitignore`)

**`migrations/`:**
- Purpose: Aerich migration history for MySQL schema
- Generated: Yes (by `aerich migrate`)
- Committed: Yes

**`static/`:**
- Purpose: Served at `/static/` path; can hold built frontend assets or other static files
- Generated: Partially (built frontend output can be placed here)
- Committed: Yes (currently contains static assets)

**`.planning/`:**
- Purpose: GSD planning documents (codebase analysis, phase plans)
- Generated: Yes (by GSD tooling)
- Committed: Yes

**`jobs_spider/*/logs/`:**
- Purpose: Rotating daily log files for standalone spider processes
- Generated: Yes (by loguru in spider scripts)
- Committed: No

---

*Structure analysis: 2026-03-21*