Sibyl: A Cybersecurity Copilot

Problem & Motivation

In 2021, the US Government Accountability Office disclosed around 2.6 million cybersecurity incidents, highlighting the escalating threats to organizations and individuals in the digital realm. This surge in digital threats poses a challenge for cybersecurity professionals who struggle to keep pace with evolving compliance standards set by institutions such as the National Institute of Standards and Technology (NIST) and the International Organization for Standardization (ISO). These standards are complex and continually evolve in response to rising cyber incidents, causing cyber fatigue among qualified professionals. This weariness impedes vigilance, thereby leaving a company’s systems vulnerable to attacks, as illustrated by the average loss of $4.45 million from a single non-compliance event. The problem is further compounded by a 21% turnover rate among professionals and a limited cybersecurity budget allocation. Recognizing the urgency of these issues, we developed Sibyl —a cutting-edge generative AI application designed to combat compliance fatigue, minimize errors, and simplify security protocol adoption, empowering cybersecurity professionals to effectively fortify systems and protect their organizations.

Minimum Viable Product (MVP)

Sibyl was constructed using cutting-edge artificial intelligence technologies. This involved integrating pre-trained Large Language Models (LLMs), vector databases, retrieval augmentation generation (RAG) systems, and parallel computing. These state-of-the-art technologies enabled us to develop a copilot adept at tackling the most challenging security inquiries. Once a document is uploaded or selected, users can perform Q&A, get a summary,  and generate a checklist of action items to meet compliance requirements. Although Sibyl was designed to accommodate various compliance document formats, the data used during her development was sourced from NIST compliance documents. 

Extensive R&D was undertaken to enhance Q&A, summary, and action checklist capabilities, leading to the discovery of key insights:

• Achieving excellent results requires refining document parsing, preparation, and storage methods tailored to specific models and tasks, surpassing the capabilities of a good LLM alone.
• How stored document content is retrieved and used alongside user queries significantly impacts responses. Fine-tuning retrieval methods and prompt engineering were pivotal for generating insightful responses.
• To support our model's cutting-edge technologies, an intricate architecture utilizing Tensor Parallelism, continuous batching, and quantization was necessary due to high computational demands.

To assess the performance of our model, we implemented a simple yet comprehensive test: gauging its ability to pass a cybersecurity exam. We chose to evaluate Sibyl on a practice test for CompTIA's globally recognized Security+ certification. Simultaneously, we scrutinized the model's generated document summary using a rubric that took into account factors such as relevance, clarity, coherence, length, and accuracy of the response. Based on this evaluation, our model achieved an impressive 82% on the exam and received an overall assessment of "Good" for the quality of its generated summary.

In a landscape marked by relentless cyber threats, Sibyl emerges as an asset, providing not only efficiency but a pivotal layer of defense for proactive and adaptive cybersecurity measures. By significantly reducing the time invested in reading compliance documents and compiling action items, Sibyl addresses cyber fatigue, allowing security professionals to regain focus and confidence. This enables them to effectively implement vital safeguards and protect the businesses they serve.

Acknowledgments

We dedicated a substantial amount of effort to developing our AI Copilot, Sibyl. Our sincere thanks go out to our families, friends, colleagues, and classmates, for their unwavering support and contributions to our success. Thank you to everyone who tested our MVP. We would also like to acknowledge and express our gratitude to the United States Air Force Cyberspace Operations and Uri Schonfeld, in addition to our capstone instructors Joyce Shen, and Todd Holloway for helping us shape our MVP. Your invaluable guidance brought our vision to life and we look forward to further developing our MVP.