Assessment Unveiled: The True Capabilities of Fireworks AI

Recently, while exploring the Fireworks.ai website, I came across some interesting information about Fireworks AI that I think is worth sharing.

In the world of artificial intelligence today, dealing with different types of data has been a big problem. Most of our data, like medical records, podcasts, and financial statements, are stored as images, PDFs, audio files, or in special knowledge bases. But traditional Large Language Models (LLMs) can’t handle these data formats very well. Even though there are Vision Language Models (VLMs) and multimodal models trying to solve this, they don’t do a great job either. They can only manage a few types of inputs and lack good reasoning skills, which means we get lower-quality results and have to pay more.

I. Introduction to Fireworks AI

Fireworks AI emerges as a revolutionary force in the AI landscape. It addresses the long-standing issue of disparate data formats that traditional LLMs struggle with. By leveraging its innovative Compound AI approach, it constructs an automated pipeline. This pipeline is designed to transform a wide array of digital asset formats into a form that is not only compatible with LLMs but also conducive to advanced processing and logical reasoning. This means that whether it’s medical records, podcasts, or financial statements, Fireworks AI aims to unlock the value within these data sources, providing a more comprehensive and intelligent solution for various applications.

II. The Game-Changing Document Inlining Feature

Document Inlining, a standout feature of Fireworks AI, is set to redefine how we handle document-based tasks. It acts as a bridge between the world of visual and textual data. This compound system has the remarkable ability to turn any LLM into a vision model. It does so by first transcribing and parsing non-textual content, such as images and PDFs, into a structured text format. This parsed data is then fed into an LLM for reasoning and further processing. With capabilities like higher quality reasoning, input flexibility for various file types, and an ultra-simple usage model, it offers a practical and efficient way to handle documents, enhancing productivity and accuracy in tasks ranging from data analysis to content extraction.

Fireworks AI is trying to change this. They want to make it easier to handle all kinds of data and get better results. Their idea is to use something called Compound AI. This means they’ve built an automatic system that can turn any digital data into a format that LLMs can understand and work with. This way, we can get high-quality results from different types of data, just like using a regular LLM.

One of the things Fireworks AI has just shown to the public is called Document Inlining. It’s a new system that can turn any LLM into a model that can read images and PDFs. This is really useful for tasks that need to work with documents.

III. Our Assessment Methodology and Model Selection

For this in-depth assessment of Fireworks AI, we decided to use the “Llama 3.3 70B Instruct” model. This choice was based on several factors. The Llama 3.3 70B Instruct model has shown impressive performance in various natural language processing tasks, making it a suitable candidate to evaluate the effectiveness of Fireworks AI’s capabilities. Its large parameter count and advanced instruction-following capabilities provide a solid foundation for testing the system’s ability to enhance reasoning and generation when dealing with different data modalities.

IV. Quality Evaluation Results and Insights

During our evaluation, we conducted a series of tests using a diverse set of data, including PDFs with complex tables and figures, as well as multiple image documents. The results were quite revealing. When compared to traditional methods of handling such data, Fireworks AI’s Document Inlining, in combination with the Llama 3.3 70B Instruct model, demonstrated significant improvements in accuracy and reasoning capabilities. For instance, in tasks that required extracting specific information from tables within PDFs, the system was able to provide more precise and reliable answers. This was attributed to the enhanced parsing and structuring capabilities of the Document Inlining feature, which allowed the Llama 3.3 70B Instruct model to better understand and process the data.

Example 1

In the above example, we input “The amount of Total revenues in Q2 - 2024” and ask it to find the corresponding value in the table. The results show that it can accurately locate the relevant data.

Example 2

In the second example, we asked it to convert the content of an image into Markdown. However, the conversion effect for formulas was not satisfactory.

Example 3

Example 4

The third and fourth examples are junior high school and high school math problems respectively, and it was required to solve them.

V. Conclusion

In this exploration of Fireworks AI, we have witnessed its potential and limitations. The Document Inlining feature is a significant step forward in handling diverse data formats, enabling LLMs to process images and PDFs more effectively. Through the use of advanced parsing techniques, it has shown the ability to extract and structure information from complex documents, as demonstrated in our example of accurately retrieving specific data from a table.

However, it is not without its drawbacks. When it comes to converting certain types of content, such as formulas in images to Markdown, the results were less than ideal. This indicates that there is still room for improvement in handling specialized and complex visual elements.

The performance on math problems, which were used as additional test cases, provided further insights. While it was able to attempt solutions for junior high and high school math questions, the accuracy and comprehensiveness of the answers varied. This suggests that its reasoning capabilities, although enhanced in some aspects, may still require fine-tuning and further development, especially in more specialized knowledge domains.

Overall, Fireworks AI shows great promise in revolutionizing the way we handle and process data, but it needs to address these identified shortcomings to reach its full potential and offer a more robust and reliable solution for a wide range of applications.