Financial AI Service Revolutionizes LLM Evaluation Indicators[1]February 24, 2024

LG CNS, a leading South Korean technology firm, has unveiled a groundbreaking financial customized artificial intelligence (AI) evaluation tool designed to help financial institutions integrate production AI effectively. This innovative tool evaluates dozens of open language models (LLM) released on the market, identifying the most suitable AI models for various financial services, including banks, insurance, and securities. Open LLMs, such as LG AI researchers’ Exaone 3.5, Meta’s LLAMA, and Alibaba’s QWEN 2.5, are models whose source code and algorithms are freely accessible to the public.[3]LG CNS staff introduces the ‘Financial Custom LLM Evaluation Tools.’
Click on the image to view it larger.

Customization and Security

Financial enterprises often face the challenge of data security when introducing AI. To address this, companies can build their own models by training AI with proprietary data, ensuring the models are tailored to their specific needs. Open LLM models provide a secure platform for businesses to develop in-house solutions without risking data leakage. Conversely, closed LLMs like OpenAI’s ChatGPT and Google’s Gemini are proprietary and do not offer the flexibility to build custom AI models.

Evaluation and Performance

LG CNS’s financial AI evaluation tool employs 29 evaluation indicators and about 1,200 datasets. The primary evaluation criteria include reasoning abilities, mathematical problem-solving, comprehension of complex questions, document summarization, financial terminology, and the proficiency of AI agents. These evaluation methods are vital in verifying the performance of AI in financial settings, particularly in addressing intricate financial inferences that AI struggles with.

Expert Collaboration and Validation

LG CNS has incorporated feedback from financial experts to ensure the evaluation metrics’ accuracy and relevance. This ensures that the AI models’ responses adhere to financial regulations and intricate service frameworks. Hyun Shin-kyun, president of LG CNS, highlighted the importance of this tool, stating,

“The biggest concern for companies that want to introduce a generated AI into financial services is to understand which AI model is best for the service. It will be the best solution to solve it quickly.”
Hyun Shin-kyun, president of LG CNS

Real-World Applications and Future Prospects

LG CNS has recently launched several AI-driven projects, including a construction AI platform with NH Nonghyup Bank and a counselor’s response system with Shinhan Card. These initiatives illustrate the practical applications of AI in the financial sector, enhancing efficiency and customer service. Revenge of the wild, California AI, provide clear examples within the U.S. A prime example in the U.S. is JPMorgan Chase’s use of AI to automate repetitive tasks, identifying over $375,000 in potential savings.

Addressing Concerns and Potential Limitations

Critics argue that the reliance on AI in financial services could lead to job displacement and overreliance on technology. However, proponents point out that AI can augment human capabilities, allowing financial professionals to focus on more strategic and client-facing tasks. Ensuring robust cybersecurity measures and ethical AI development are also crucial to mitigate risks and build public trust.

Conclusion

LG CNS’s AI evaluation tool represents a significant stride in the financial technology sector, providing a comprehensive framework for evaluating and deploying AI models. This tool not only ensures that financial institutions can integrate production AI effectively but also paves the way for more innovative and customer-centric financial services.]

Q&A on Financial AI Service Revolutionizing LLM Evaluation Indicators

What is LG CNS’s new AI evaluation tool, and how does it assist financial institutions?

LG CNS has introduced a pioneering financial customized Artificial Intelligence (AI) evaluation tool designed too streamline the integration of production AI in financial services. This tool evaluates numerous open language models (LLMs) available in the market, pinpointing the most fitting AI models for various financial sectors like banking, insurance, and securities. By doing so,it aids financial institutions in selecting the appropriate AI systems that align with their specific needs,enhancing service efficiency and innovation. The tool leverages readily accessible open LLMs such as Exaone 3.5, LLAMA, and QWEN 2.5, whose source codes and algorithms are open to the public[[[1] [2]].

How do open LLMs provide a secure platform for financial enterprises?

Open llms offer financial businesses a secure environment to develop customized AI models without the risk of data exposure. By allowing companies to train AI with their proprietary data, open LLMs ensure that the models are specifically tailored to their requirements. This tailored approach contrasts with closed LLMs, like ChatGPT and Google’s Gemini, which are proprietary and lack customization flexibility. Open-source models mitigate data leakage risks, thus ensuring greater data security for enterprises[[[1]].

What are the evaluation criteria used by LG CNS’s AI tool?

To ensure the AI models’ effectiveness in financial settings, LG CNS’s tool employs 29 evaluation indicators and approximately 1,200 datasets. The key criteria include:

• Reasoning Abilities: Evaluates the model’s capability to engage in logical decision-making processes.
• Mathematical Problem-Solving: Tests the model’s proficiency in solving complex mathematical problems.
• Comprehension of Complex Questions: Measures the ability to interpret and answer intricate queries.
• Document Summarization: Assesses the model’s capability to provide concise summaries of extensive documents.
• Financial Terminology: Checks the model’s understanding and use of industry-specific jargon.
• Proficiency of AI Agents: Evaluates the overall efficiency and skill level of AI agents in executing tasks[[[3]].

Why is expert collaboration crucial in the advancement of financial AI tools?

involving financial experts in the development of evaluation metrics is essential for ensuring their accuracy and relevance. This collaboration guarantees that AI models’ responses comply with financial regulations and align with the complex frameworks of financial services. By verifying these aspects, LG CNS ensures the tool’s reliability and applicability, reinforcing its value to the industry. Hyun Shin-kyun, president of LG CNS, emphasizes the importance of swiftly identifying the most suitable AI models for financial services to bridge the existing gap[[[3]].

How are AI-driven initiatives being implemented in practical financial applications?

Recent AI-driven projects by LG CNS demonstrate the practical applications of AI in the financial sector. These include a construction AI platform with NH Nonghyup Bank and a response system for counselors with Shinhan Card, designed to enhance efficiency and customer service. In the U.S., JPMorgan Chase’s AI initiative exemplifies larger-scale applications, where AI has automated repetitive tasks, leading to potential savings of over $375,000. Such implementations illustrate AI’s power to streamline operations and improve service quality[[[1]].

What are the concerns and benefits of integrating AI in financial services?

Critics of AI integration in financial services argue that it could lead to job displacement and an overreliance on technology.However, supporters highlight that AI can complement human skills, allowing professionals to focus more on strategic and client-centered tasks. Ensuring robust cybersecurity and ethical AI development is vital to mitigate such risks and foster public trust. Thus, the integration of AI, if managed carefully, can result in a balanced approach to technological advancement in the financial sector[[