TaskMiner

Problem & Motivation

Although communication at work has evolved over time, one medium remains a mainstay: email. According to McKinsey, as of 2012, the average professional spends 28% of the workweek managing email (source). A study commissioned by Slack in 2019 revealed that workers in the US and Britain draft over 110 emails per week (source). And based on our user survey for this project, 50% of respondents consider email their main communication tool for work.

Buried in this trove of unstructured data are tasks: requests for updated slides, inquiries about project staffing, invitations to coffee chats, and millions of other industry-specific to-dos. These tasks are sometimes lost in the avalanche of emails a worker receives daily, leading to missed deadlines and disappointed stakeholders.

Some workers use ticketing systems, such as Jira, to stay organized. However, these systems can be too formal or robust for many users’ needs. Other existing solutions, such as Notion or Trello, are too open-ended or require burdensome manual input.

TaskMiner provides a lightweight AI-powered solution for extracting tasks from emails. The extraction process is fully automated, without the need for manual prompting. Generated tasks are compiled in a simple dashboard for users to review and mark as completed. Through this simple workflow, TaskMiner’s mission is to provide a task tracking solution for professionals who live in email, but do not have a systemized process for handling incoming requests.

How to use

To use TaskMiner, users need to follow these simple steps:

1. Log-in with their credentials to the front-end website
2. In the Dashboards page, scroll down to the Email Upload section and upload a batch of emails to process
3. Wait for the page to refresh after emails have been processed
4. Sort by “Identified Date Descending” to view the latest identified tasks first
5. Accept or Decline the identified tasks
6. View accepted tasks on the Task Manager dashboard and mark them as complete when done

Data Science Approach

Emails are not independent of one another: they are grouped into threads, with different threads often relating to the same subject. These various emails within and across threads often provide useful context for a task.

Therefore, TaskMiner uses a Retrieval-Augmented Generation (RAG) pipeline to (1) assess whether an email contains a task, (2) query relevant context to that task from other emails, (3) assess whether that task is already complete or duplicative of another task, and (4) generate a to-do item for outstanding tasks.

More specifically, the tool is implemented as follows:

• Email batches are ingested through a website portal and uploaded to the user’s folder in an S3 bucket.
• Uploaded emails are parsed and stored into an Emails table on a DynamoDB database. These parsed emails are then stored in a Pinecone vector store along with email metadata.
• For each email, an LLM is queried to identify whether it contains a task. For each email with a task, the vector store is queried to retrieve related emails that supplement context for that task.
• If the original email or queried context indicates the task is already complete or duplicative of another task, it is marked as such and TaskMiner continues to the next email.
• If the task is new, using the queried context, the task is summarized by the LLM, including task metadata such as requestor, due date, and links to related emails.
• Tasks are stored in a Tasks table, through which they can be displayed on a Streamlit-based front-end (i.e., the website) for the user to review.
• Approved emails are tracked using the Task Manager portal on the website, where the user can mark them as complete.

This high-level implementation is summarized in the image below:

Architecture Summary

Data Sources

To develop and evaluate the model, we relied on two datasets: (1) the Enron email corpus; and (2) synthetically generated data.

Enron was an energy company that went bankrupt in 2002 after infamously committing accounting fraud. During Enron’s bankruptcy proceeding, the Federal Energy Regulatory Commission released approximately 500,000 emails belonging to about 150 senior Enron employees. These emails are often studied in computer science and machine learning projects.

We selected the latest six months of emails in the data belonging to five employees with a significant number of emails. For approximately 30-50 emails belonging to each user, we manually labeled “golden truths”: whether the emails contained tasks, if those tasks are duplicative/completed, and the task description. We used these emails to assist in development of the RAG pipeline and for final evaluation.

The Enron data provided us with real-world professional emails with which to conduct experimentation. However, there are several downsides to these data: first, Enron emails are from 2002, and communication has changed significantly since then. Second, Enron emails are available only for senior management; therefore, their generalizability could potentially be more limited. Finally, it was difficult to easily identify examples of complex interconnected email threads in the Enron data.

Therefore, we supplemented our development and evaluation datasets with synthetic data. We generated seven different email types, including “single thread with a task,” “self reminder with a task,” “multiple threads with no task,” etc. We then leveraged both AI and manually-written email threads to generate synthetic emails with which to develop and test our data. We used one dataset containing approximately 35 synthetic emails for development and another dataset containing approximately 60 emails for evaluation.

Model Evaluation

The RAG pipeline described above relies on a variety of hyperparameters, including the prompts provided to the LLMs, the LLM itself, the function for sorting through related emails in the vector store, and more.

To tune the hyperparameters appropriately, we relied on a three-pronged evaluation approach. Throughout this approach, we prioritized keeping a task on the user’s dashboard to mark as complete, to minimize the chance of a task falling between the cracks. The evaluation approach is as follows:      

1. Evaluate whether the email contains a task
   1. Relevant metric: recall (prioritize minimizing false negatives)
2. Retriever performance
   1. Relevant metric: Mean F1 score (balances retrieving relevant emails with avoiding unrelated context that reduces accuracy)
3. Task summarization
   1. Relevant metric: RAGAS context precision, faithfulness, relevancy, answer similarity (weighted 5%, 15%, 40%, and 40%, respectively)

To assess evaluation overall, we generated a combined metric that weights the above metrics as follows: 40% weight on prong 1; 20% weight on prong 2; 40% weight on prong 3. For Enron emails, where we did not have golden truths for relevant emails, we excluded evaluation of the retriever performance and RAGAS context precision.

Key Learnings & Impact

In developing TaskMiner, we learned that more expensive LLMs do not guarantee better performance. Instead, clear task definitions and prompt engineering - particularly few-shot prompts guiding response length and style - are critical.

Retrieval thresholds must balance noise and relevance, as low thresholds introduce spurious context while high thresholds risk omitting useful information. Most emails are self-contained and do not require extra context, thus simplifying retrieval. TaskMiner’s lightweight, deterministic RAG solution bridges the gap between heavyweight project management systems and manual email workflows for professionals who “live in email,” and its front-end click-through capability enables direct navigation to related emails, reducing cognitive load.

Future Work

Improving task identification: Emails are an inherently complex data source. Different users have different communication styles. Relationships among employees contain implicit information about whether an email contains a task or does not. Therefore, a lot of “grey area” exists in the space of task identification.

For this reason, many of our future development ideas relate to fine-tuning the task identification process. Here is a sampling of those ideas:

• Generate a small number of sample “grey area” emails for the user to review and categorize as “tasks” or “not tasks.” Based on their categorization, personalize task recommendations to be more expansive or stricter.
• Using previously uploaded emails, assess the individuals most often communicated with. For each of those, ask the user to describe their relationship with them (e.g., “I am their Executive Assistant”) to assist with task categorization.
• Add learning features for each user based on their categorization of certain tasks as “tasks” or “not tasks.” For example, if a user consistently rejects a task responding to calendar invitations, do not generate tasks corresponding to invitations.

Dynamic inboxes: To-do lists and inboxes are both dynamic in nature. As additional emails are sent, certain tasks are crossed off the list and replaced by follow-ups. Our MVP handles these dynamics through the RAG pipeline for a defined set of emails (e.g., the 200 emails received this week), but it is not currently set up to address a dynamic inbox with newly-arriving emails. Incorporating this feature would increase TaskMiner’s viability for deployment in a real-world professional setting.

Further integration: Our MVP uses a web interface to ingest emails and extract tasks from them. In the future, we would like to incorporate the tool into enterprise email systems, such as Outlook, Gmail, and mobile applications.

Acknowledgements

• Our friends, family, and colleagues who participated in our user interviews and survey
• Joyce Shen and Amit Bhattacharyya, our Capstone instructors who provided guidance and feedback throughout
• Mark Butler, and many of our professor in DATASCI 267, for teaching us about LLMs and RAG systems