aic_seminar_2026

<section>
                <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/Vertical_Logo_AIC_White.png" alt="AIC Logo" style="position: absolute; top: -50px; left: -100px; height: 100px; margin: 0; background: transparent; border: none; box-shadow: none;">
                <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/StratoLogo.png" alt="Stratosphere Logo" style="position: absolute; top: -50px; right: -100px; height: 100px; margin: 0; background: transparent; border: none; box-shadow: none;">
                <h2 style="margin-top: 50px;">Abstracting State and Action Spaces in Network Security Games via LLM Embeddings</h2>
                <p style="color:MediumSeaGreen;">Maria Rigaki</p>
                <p>Stratosphere Research Laboratory, AI Center</p>
                <p style="position: absolute; bottom: -50px; right: -10px; font-size: 0.6em;">January 28th, 2026</p>
            </section>

            <!-- Slide 2: Context -->
            <section>
                <section>
                    <h2>RL in Cybersecurity</h2>
                    <ul style="text-align: left; width: 90%; margin: auto;">
                        <li><strong>Autonomous</strong> agents defending or attacking complex networks</li>
                        <li>The <strong>environment</strong> abstracts the real network</li>
                        <!-- <li><strong>Current Approach:</strong> Vectorized representations of state/action spaces.</li> -->
                    </ul>
                </section>
                <section>
                    <h3>Reinforcement Learning Loop</h3>
                    <img src="https://deepsense.ai/wp-content/uploads/2023/02/Figure-2-Classic-reinforcement-learning-training-loop.png" height="400" data-preview-image>
                    <!-- <div class="placeholder-box">
                    [PLACEHOLDER: RL Loop Diagram (Agent <-> Environment)]
                </div> -->
                </section>
                <section>
                    <h3>Data Exfiltration Scenario</h3>
                    <img src="https://stratosphereips.github.io/NetSecGame/figures/scenarios/scenario_1.png" height="550" data-preview-image>
                </section>
            </section>

            <!-- Slide 3: The Problem -->
            <section>
                <section>
                    <h2>How to Represent States?</h2>
                    <ul>
                        Reinforcement Learning requires vectors for state and action spaces.
                    </ul>
                     
                </section>
                <section>
                    <h2>What are the problems with this?</h3>
                        <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/vectorized_state.jpg" data-preview-image>
                </section>
                <section>
                    <h3>1. Information Leakage</h3>
                    <p>Fixed vectors often inadvertently expose privileged information to the agent.</p>
                </section>
                <!-- <div style="text-align: left;"> -->
                <section>
                    <h3>2. Scalability Issues</h3>
                    <p>Fixed-size input fails when network topology changes or scales up (e.g., adding new hosts).</p>
                </section>
                    
                <!-- </div> -->
            </section>

            <!-- Slide 4: NetSecGame -->
             <section>
            <section>
                <h2><a href="https://github.com/stratosphereips/NetSecGame/">NetSecGame</a></h2>
                <ul style="text-align: left;">
                    <li>Shifts from fixed vectors to <strong>flexible set representations</strong>.</li>
                    <li>Defines state via sets: (Networks, Hosts, Services, Data, etc.)</li>
                    <li>The agent decides how to use the information</li>
                </ul>
                <!-- <div class="placeholder-box">
                    [PLACEHOLDER: NetSecGame Scenario Topology]
                </div> -->
            </section>

            <!-- Slide 4.5: State Representation Example -->
            <section>
                <h3>Example State Representation</h3>
                <pre data-id="code-animation"><code class="language-json" data-trim>
    {"known_networks": [
        {"ip": "192.168.1.0", "mask": 24}, 
        {"ip": "213.47.23.192", "mask": 26}], 
    "known_hosts": [
        {"ip": "213.47.23.195"}, 
        {"ip": "192.168.1.2"}], 
    "controlled_hosts": [
        {"ip": "213.47.23.195"}, 
        {"ip": "192.168.1.2"}], 
    "known_services": {},
    "known_data": {},
    "known_blocks": {}}
				</code></pre>
                <!-- <p>Unlike fixed vectors, we use structured data (JSON) to handle arbitrary topologies.</p>
                <div class="placeholder-box">
                    [PLACEHOLDER: JSON State Object Example]
                </div> -->
            </section>
            <section>
                <h2>Action Space</h2>
                <ul>
                    <li>5 action types: <strong>ScanNetwork, ScanServices, ExploitService, FindData, ExfiltrateData</strong></li>
                    <li>Actions are parameterized</li>
                    <li>The action space is not fixed. The number of valid actions increases as the agent discovers new objects.</li>
                </ul>
                
            </section>
            <section>
                <h3>Example Action Representation</h3>
                <pre data-id="code-animation"><code class="language-json" data-trim>
{
    "action": "ScanNetwork",
    "parameters": {
        "target_network": "192.168.1.0/24", 
        "source_host": "192.168.1.2"}
}
                </code></pre>
            </section>
            
        </section>
            <!-- Slide 5: LLM Embeddings -->
        <section>
            <section data-auto-animate>
                <h2>Why LLM Embeddings?</h2>
            </section>
            <section data-auto-animate>
                <h2>Why LLM Embeddings?</h2>
                <ul>
                    <li>Frontier and local LLMs worked well as agents in the past</li>
                </ul>
            </section>
            <section data-auto-animate>
                <h2>Why LLM Embeddings?</h2>
                <ul>
                    <li>Frontier and local LLMs worked well as agents in the past</li>
                    <li>Semantically meaningful latent representations</li>
                </ul>
            </section>
            <section>
                <h2>DQN Agent Architecture</h2>
                <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/q_net.png" data-preview-image>
            </section>
        </section>
            <!-- Slide 6: Analysis of Existing Models -->
            <section>
                <section>
                    <h2>Base model</h2>
                    <ul>
                        <li><a href="https://huggingface.co/Qwen/Qwen3-Embedding-0.6B">Qwen3-Embedding-0.6B</a></li>
                        <li>Context window: 32K</li>
                        <li>Embedding size: 1024</li> 
                    </ul>
                </section>
                <section>
                    <h2>Win Rate</h2>
                    <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/win_rate.png" height="500" data-preview-image>
                </section>
                <section>
                    <h2>The problem</h2>
                    <p>Changing to a new IP range and the win rate drops to 0 :(</p>
                </section>

                <section data-background-color="white">
                        <!-- <h2>UMAP representation</h2> -->
                        <img src="https://raw.githubusercontent.com/MariaRigaki/figures/refs/heads/main/umap.png" height="720" data-preview-image>
                </section>
            </section>
            <section>
                <h2>What to do?</h2>
                <ul>
                    <li>Try different models: CodeBert, nomic-embed-text-v1</li>
                    <li>Try different state representations: JSON, Text, etc</li>
                    <li>Fine-tuning?</li>
                </ul>
            </section>
            <!-- Slide 7: Fine-Tuning -->
            <section>
                <h2>Fine-Tuning</h2>
                <ul style="text-align: left;">
                    <li><strong>Goal:</strong> Fix limitations of off-the-shelf models.</li>
                    <li><strong>Dataset Creation:</strong> Generating domain-specific training pairs / triplets.</li>
                    <li><strong>Objective:</strong> Capture <strong>structural similarities</strong>.</li>
                </ul>
                
            </section>

            <!-- Slide 8: Dataset generation -->
        <section>
            <section data-auto-animate data-auto-animate-unmatched="fade">
                <h3>Dataset Generation</h3>
                Create a structure map that maps a specific IP to a string:
                    <pre data-id="state"><code class="language-json" data-trim>
    {"known_networks": [
        {"ip": "192.168.1.0", "mask": 24}, 
        {"ip": "213.47.23.192", "mask": 26}], 
    "known_hosts": [
        {"ip": "213.47.23.195"}, 
        {"ip": "192.168.1.2"}], 
    "controlled_hosts": [
        {"ip": "213.47.23.195"}, 
        {"ip": "192.168.1.2"}], 
    "known_services": {},
    "known_data": {},
    "known_blocks": {}}
				</code></pre>
                <!-- </div> -->
            </section>
            <section data-auto-animate data-auto-animate-unmatched="fade">
                <h3>Dataset Generation</h3>
                Create a structure map that maps a specific IP to a string:
                    <pre data-id="state"><code class="language-json" data-trim>
    {"known_networks": [
        {"ip": "IP", "mask": 24}, 
        {"ip": "IP", "mask": 26}], 
    "known_hosts": [
        {"ip": "IP"}, 
        {"ip": "IP"}], 
    "controlled_hosts": [
        {"ip": "IP"}, 
        {"ip": "IP"}], 
    "known_services": {},
    "known_data": {},
    "known_blocks": {}}
				</code></pre>
                <!-- </div> -->
            </section>
            <section data-auto-animate>
                <p>States with the same structures are "positive" pairs</p>
            </section>
            <section data-auto-animate>
                <p>States with the same structures are "positive" pairs</p>
                <p>States with different structures are "negative" pairs</p>
            </section>
        </section>
        <section>
            <section data-auto-animate>
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss">Multiple Negatives Ranking Loss</a></h2>
                <ul>
                    <li>Triplet dataset: anchor, positive, negative (optional)</li>
                </ul>
            </section>
            <section data-auto-animate>
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss">Multiple Negatives Ranking Loss</a></h2>
                <ul>
                    <li>Triplet dataset: anchor, positive, negative (optional)</li>
                    <li>Given an anchor assign the highest similarity to the corresponding positive out all positive and negatives in the batch.</li>
                </ul>
            </section>
            <section data-auto-animate="">
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss">Multiple Negatives Ranking Loss</a></h2>
                <ul>
                    <li>Triplet dataset: anchor, positive, negative (optional)</li>
                    <li>Given an anchor assign the highest similarity to the corresponding positive out all positive and negatives in the batch.</li>
                    <li>The model must pick the correct positive.</li>
                </ul>
            </section>
            <section data-auto-animate="">
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss">Multiple Negatives Ranking Loss</a></h2>
                <ul>
                    <li>Triplet dataset: anchor, positive, negative (optional)</li>
                    <li>Given an anchor assign the highest similarity to the corresponding positive out all positive and negatives in the batch.</li>
                    <li>The model must pick the correct positive.</li>
                    <li>The harder this “picking” is, the stronger the model will become</li>
                </ul>
            </section>
        </section>
        <!-- Slide 10: OnlineContrastiveLearning -->
        <section>
            <section data-auto-animate>
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#contrastiveloss">Contrastive Loss</a></h2>
                <ul>
                    <li>Requires dataset pairs and binary labels</li>
                </ul>
            </section>
            <section data-auto-animate>
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#contrastiveloss">Contrastive Loss</a></h2>
                <ul>
                    <li>Requires dataset pairs and binary labels</li>
                    <li>Positive pairs: label = 1, Negative pairs: label = 0</li>
                </ul>
            </section>
            <section data-auto-animate>
                <h2><a href="https://sbert.net/docs/package_reference/sentence_transformer/losses.html#contrastiveloss">Contrastive Loss</a></h2>
                <ul>
                    <li>Requires dataset pairs and binary labels</li>
                    <li>Positive pairs: label = 1, Negative pairs: label = 0</li>
                    <li>Margin: Negative samples (label == 0) should have a distance of at least the margin value.</li>
                    <!-- <li>Computes loss only for hard positives or hard negatives.</li> -->
                </ul>
            </section>
        </section>
		<!-- Slide 11: metrics and results -->
            <section>
                <section>
                    <h2>Sensitivity Metric</h2>
                    <ul>
                        <li>Take an example state</li>
                        <li>Change the IPs (No structural change)</li>
                        <li>Add a new known host (Minor structural change)</li>
                        <li>Add new data (Major structural change)</li>
                    </ul>
                </section>
                <section>
                    <h2>Some Results</h2>
                    <table style="font-size: 0.7em">
						<thead>
							<tr>
								<th>Model</th>
								<th>S1 (&gt0.98)</th>
								<th>S2 (&lt0.90)</th>
                                <th>S3 (&lt0.70)</th>
							</tr>
						</thead>
						<tbody>
							<tr>
								<td>Qwen3-Embedding-0.6B (text)</td>
								<td>0.9181</td>
								<td>0.9987</td>
                                <td>0.9801</td>
							</tr>
                            	<tr>
								<td>Qwen3-Embedding-0.6B (JSON)</td>
								<td>0.9584</td>
								<td>0.9973</td>
                                <td>0.9714</td>
							</tr>
							<tr>
								<td>Finetuned (MNRL-JSON)</td>
								<td>0.8386</td>
								<td>0.5017</td>
                                <td>0.0952</td>
							</tr>
                            <tr>
                                <td>Finetuned (OCL-Text)</td>
								<td>0.7233</td>
								<td>0.9432</td>
								<td>0.5551</td>
							</tr>
						</tbody>
					</table>
                </section>
            </section>
            <!-- Slide 12: Findings -->
            <section>
                <h2>Findings (so far)</h3>
                    <ul>
                        <li>The base model does not understand the data structure</li>
                        <li>Text or JSON don't seem to make a big difference</li>
                        <li>MultipleNegativesRankingLoss shows better (?) performance</li>
                    </ul>
            </section>
            <!-- Slide 13: Next Steps -->
            <section>
                <h2>Next Steps</h3>
                    <ul>
                        <li>Increase the number of sensitivity tests (robustness and coverage)</li>
                        <li>Augment the dataset (harder negatives)</li>
                    </ul>
            </section>
            <section><h1>Thank you!</h1></section>
			<section>
                <h2>Links</h2>
                <p><a href="https://github.com/stratosphereips/NetSecGame">https://github.com/stratosphereips/NetSecGame</a></p>
                <p><a hred="https://huggingface.co/stratosphere/datasets">https://huggingface.co/stratosphere/datasets</a></p>
            </section>