CY0-001 Exam Notes

Prompt injection = attacker inserts instructions to make the model perform UNINTENDED ACTIONS (e.g., exfiltrate data, ignore system prompt, call unauthorized APIs). Jailbreaking = attacker crafts inputs specifically to BYPASS SAFETY GUARDRAILS (e.g., produce restricted content, disable content filtering). Injection is about unauthorized actions; jailbreaking is about bypassing restrictions. Both manipulate prompts but have different attack goals and different primary mitigations (prompt firewalls vs. guardrail hardening).

Model inversion TARGET = the TRAINING DATA. Attacker queries the model to reconstruct sensitive data used during training (e.g., PII, medical records). Model theft TARGET = the MODEL ITSELF. Attacker queries the model repeatedly to build a functional clone. Both use repeated querying but the stolen asset is completely different. Rate limiting helps mitigate both but is not the only control for either.

Data poisoning is an ATTACK on the model (corrupt training data to degrade/manipulate model behavior). Membership inference is a PRIVACY LEAK (determine if a specific record was used in training — useful to confirm someone's presence in a sensitive dataset). Poisoning changes what the model does. Membership inference reveals what data was used.

Direct injection = malicious instructions in the USER'S PROMPT input, directly targeting the model interface. Indirect injection = malicious instructions EMBEDDED IN EXTERNAL DATA the model retrieves and processes (e.g., a document retrieved via RAG, a web page being summarized). Indirect injection is harder to prevent because the attacker doesn't need direct model access — they only need to control content the model reads.

LLM Top 10 (2025) = APPLICATION-LAYER risks for systems built with LLMs: prompt injection (LLM01), sensitive info disclosure (LLM02), supply chain (LLM03), data and model poisoning (LLM04), improper output handling (LLM05), excessive agency (LLM06), system prompt leakage (LLM07), vector and embedding weaknesses (LLM08), misinformation (LLM09), unbounded consumption (LLM10). ML Security Top 10 (2023) = MODEL/DATA-LAYER risks for ML systems broadly: input manipulation (ML01), data poisoning (ML02), model inversion (ML03), membership inference (ML04), model theft (ML05), AI supply chain (ML06), transfer learning attack (ML07), model skewing (ML08), output integrity attack (ML09), model poisoning (ML10). Key trap: model DoS and overreliance appear in the 2023 LLM list but NOT in the 2025 LLM list or the ML Security Top 10. Know which list each specific risk belongs to.

Jailbreaking is the broad category of bypassing model safety restrictions. Guardrail circumvention is the specific technique of exploiting edge cases, encoding tricks, or multi-step attacks to bypass programmatic guardrail rules. Jailbreaking can bypass both the model's built-in behavior AND external guardrails. Guardrail circumvention specifically targets the guardrail control layer.

Misinformation = false information spread WITHOUT INTENT TO DECEIVE. The person spreading it may believe it is true. Disinformation = false information spread WITH DELIBERATE INTENT to deceive, manipulate, or cause harm. The exam uses this distinction in deepfake contexts. A CEO deepfake video spread intentionally to manipulate a stock price = disinformation. AI-generated incorrect medical advice shared innocently = misinformation.

Automated pen testing = OFFENSIVE tool that ACTIVELY PROBES for vulnerabilities, exploits attack paths, generates findings. Anomaly detection = DEFENSIVE tool that PASSIVELY MONITORS for behavioral deviations from baseline. Pen testing requires authorization to run; anomaly detection runs continuously. Questions about 'identifying unusual patterns without known signatures' = anomaly detection. Questions about 'probing systems for exploitable flaws' = automated pen testing.

AI-enhanced social engineering = BROAD category using AI to craft more convincing phishing emails, context-aware pretexting, and personalized lures at scale. Deepfake impersonation = SPECIFIC TECHNIQUE using AI-generated synthetic media to impersonate a real, specific individual. Deepfake impersonation is one type of AI-enhanced social engineering. If the question describes a fake video/audio of a named person, the answer is deepfake impersonation specifically.

EU AI Act = LEGALLY BINDING REGULATION with risk-based classification and financial penalties for non-compliance. Applies to AI deployed or affecting EU residents regardless of where the developer is located. NIST AI RMF = VOLUNTARY FRAMEWORK providing structured guidance for managing AI risks. No enforcement, no penalties, no mandatory compliance. If a question asks about legal obligations, the answer involves EU AI Act. If a question asks about risk management best practices, NIST AI RMF is more appropriate.

AI Risk Analyst = PROACTIVE role — identifies, assesses, and quantifies risks BEFORE and DURING AI deployment. Works with development teams. AI Auditor = REACTIVE/VERIFICATION role — independently verifies AI systems for compliance, fairness, and accuracy AFTER deployment. Works independently from development. Risk analysts prevent problems; auditors verify outcomes.

Sanctioned AI = models and tools APPROVED by the organization, subject to security review, data governance, and policy controls. Unsanctioned AI = Shadow AI, used without IT/security approval. Key risks of unsanctioned AI: company data sent to external AI providers with unknown retention policies, no monitoring of AI outputs, potential regulatory violations. The exam tests whether you can identify Shadow AI scenarios and understand why they are governance failures.

Both are responsible AI principles but at different levels. Transparency = the organization is OPEN ABOUT WHETHER AND HOW AI is used (users know they are interacting with AI). Explainability = the AI's DECISIONS CAN BE UNDERSTOOD by humans (why did the model reach this outcome). Transparency is about disclosure; explainability is about interpretability.

LLMs = billions of parameters, general-purpose capability, high computational cost, broad knowledge. SLMs = fewer parameters, task-specific, lower cost, deployable on edge devices or with limited compute. Exam scenarios that mention cost constraints, edge deployment, privacy requirements (keeping data local), or specific narrow tasks point toward SLMs as the better choice.

Fine-tuning = PERMANENTLY MODIFIES the model by training on new data. Changes persist across all future queries. Costly in compute and time. RAG = TEMPORARILY AUGMENTS the context window with retrieved documents. No model modification. Applied at query time. If a question asks about providing up-to-date or proprietary information WITHOUT modifying the model, the answer is RAG. If the question asks about specializing a model for a domain permanently, the answer is fine-tuning.

Supervised = LABELED data with known correct answers. Used for classification (spam/not-spam) and regression. Unsupervised = UNLABELED data, model discovers patterns. Used for clustering, anomaly detection, dimensionality reduction. Reinforcement learning = REWARD/PENALTY feedback from environment, no pre-labeled dataset. Used for sequential decision-making, RLHF (Reinforcement Learning from Human Feedback). The exam tests all three and their appropriate use cases.

Pruning = REMOVES parameters (weights/neurons) from the model. Reduces parameter COUNT. Quantization = REDUCES PRECISION of existing parameters (e.g., 32-bit float to 8-bit int). Reduces storage SIZE. Both shrink models, but pruning changes the architecture while quantization only changes the representation. Exam question: 'reducing 32-bit to 8-bit' = quantization. 'Removing unnecessary weights' = pruning.