Recent studies have demonstrated that Large Language Models (LLMs) can perform various Knowledge Graph-related tasks, including Knowledge Graph Construction, even in Zero- and Few-Shot settings. However, LLMs are prone to hallucinating information and producing non-deterministic outputs, which can result in flawed reasoning, even when the answers appear to meet user expectations. This unpredictability limits their integration into automated natural language processing pipelines, such as those used in chatbots or Task-Oriented Dialogue systems. To explore the potential and limitations of LLMs in Knowledge Graph tasks, we evaluate three prominent models, namely Mixtral-8x7b-Instruct-v0.1, GPT-3.5-Turbo-0125, and GPT-4o, on constructing static knowledge graphs. Our approach uses prompts based on the TELeR taxonomy in Zero- and One-Shot sce- narios, within the context of a Task-Oriented Dialogue system. We also propose a flexible evaluation framework that captures all usable information generated by the models, alongside traditional strict metrics, and introduce TODSet, a dataset tailored to gauge the performance of LLMs on knowledge graph-related tasks. Our findings suggest that, with well-designed prompts containing sufficient detail and examples, LLMs can effectively contribute to Knowledge Graph Construction tasks.

To reproduce the experiments, follow the guidelines in the notebook. All other references to previous works of ours can be found on Github.